[otb-users] using all cores with multithreaded filters

Discussion:

Julien Radoux

2018-09-13 09:09:16 UTC

We have a few custom filters that use ThreadedGenerate data. We didn't
realize immediately, but it seems that the parallel computing is not
working properly with our new configuration :

openSuse 15.0, OTB 6.7 (git), gcc 7.3.1

When we launch a filter with ThreadedGenerateData, the number of thread is
well equal to the number of core (8 threads for 8 cores in my case), but
it only uses 1. This is not the case when we use otbcli_applications.

Diagnostic:
$ taskset -p $(pgrep LWSmoothing)
pid 29938's current affinity mask: 1
$ htop # => process total: 100%, 8 sub-processes: 12%

If we specify before, it does not change anything
$ taskset ff /usr/local/lifewatch/tools/LWSmoothing ...
$ taskset -p $(pgrep LWSmoothing)
pid 30241's current affinity list: 0

If it specify after, it works as expected (800%)
$ taskset -p ff $(pgrep LWSmoothing)
pid 30312's current affinity mask: 1
pid 30312's new affinity mask: ff
$ htop # => ok, 800%

Does anyone have an idea about the reason for this difference ? Is there
something to change in the cxx, the txx or in the compiler's configuration ?

Thanks,

Julien
--
--
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to otb-***@googlegroups.com
To unsubscribe from this group, send email to
otb-users+***@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
---
You received this message because you are subscribed to the Google Groups "otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Poughon Victor

2018-09-14 16:01:03 UTC

Permalink

Hi Julien,

Did you try:

export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=8

This is documented in https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Documentation/Cookbook/rst/AdvancedUse.rst along with other environment variables that affect OTB.

Victor Poughon

De : otb-***@googlegroups.com <otb-***@googlegroups.com> De la part de Julien Radoux
EnvoyÃ© : jeudi 13 septembre 2018 11:09
Ã : otb-users <otb-***@googlegroups.com>
Objet : [otb-users] using all cores with multithreaded filters

We have a few custom filters that use ThreadedGenerate data. We didn't realize immediately, but it seems that the parallel computing is not working properly with our new configuration :

openSuse 15.0, OTB 6.7 (git), gcc 7.3.1

When we launch a filter with ThreadedGenerateData, the number of thread is well equal to the number of core (8 threads for 8 cores in my case), but it only uses 1. This is not the case when we use otbcli_applications.

Diagnostic:
$ taskset -p $(pgrep LWSmoothing)
pid 29938's current affinity mask: 1
$ htop # => process total: 100%, 8 sub-processes: 12%

If we specify before, it does not change anything
$ taskset ff /usr/local/lifewatch/tools/LWSmoothing ...
$ taskset -p $(pgrep LWSmoothing)
pid 30241's current affinity list: 0

If it specify after, it works as expected (800%)
$ taskset -p ff $(pgrep LWSmoothing)
pid 30312's current affinity mask: 1
pid 30312's new affinity mask: ff
$ htop # => ok, 800%

Does anyone have an idea about the reason for this difference ? Is there something to change in the cxx, the txx or in the compiler's configuration ?

Thanks,

Julien

--
--
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to otb-***@googlegroups.com<mailto:otb-***@googlegroups.com>
To unsubscribe from this group, send email to
otb-users+***@googlegroups.com<mailto:otb-users+***@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
---
You received this message because you are subscribed to the Google Groups "otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+***@googlegroups.com<mailto:otb-users+***@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout.
--
--
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to otb-***@googlegroups.com
To unsubscribe from this group, send email to
otb-users+***@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
---
You received this message because you are subscribed to the Google Groups "otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Julien R

2018-09-17 20:06:29 UTC

Permalink

Hi Victor,

Thanks for your answer. I've tried it but my problem is that it does not
fully use the threads (8 thread -> 12.5% on each thread, and I would like
100% for each thread)

Post by Julien Radoux
We have a few custom filters that use ThreadedGenerate data. We didn't
realize immediately, but it seems that the parallel computing is not
openSuse 15.0, OTB 6.7 (git), gcc 7.3.1
When we launch a filter with ThreadedGenerateData, the number of thread
is well equal to the number of core (8 threads for 8 cores in my case),
but it only uses 1. This is not the case when we use otbcli_applications.
$ taskset -p $(pgrep LWSmoothing)
pid 29938's current affinity mask: 1
$ htop # => process total: 100%, 8 sub-processes: 12%
If we specify before, it does not change anything
$ taskset ff /usr/local/lifewatch/tools/LWSmoothing ...
$ taskset -p $(pgrep LWSmoothing)
pid 30241's current affinity list: 0
If it specify after, it works as expected (800%)
$ taskset -p ff $(pgrep LWSmoothing)
pid 30312's current affinity mask: 1
pid 30312's new affinity mask: ff
$ htop # => ok, 800%
Does anyone have an idea about the reason for this difference ? Is there
something to change in the cxx, the txx or in the compiler's configuration ?
Thanks,
Julien

--
--
Check the OTB FAQ at
http://www.orfeo-toolbox.org/FAQ.html

You received this message because you are subscribed to the Google
Groups "otb-users" group.
To post to this group, send email to otb-***@googlegroups.com
To unsubscribe from this group, send email to
otb-users+***@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/otb-users?hl=en
---
You received this message because you are subscribed to the Google Groups "otb-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Julien Radoux

2018-10-03 11:43:39 UTC

Permalink

With the help of my colleague, we found a workaround to the problem. It
seems to be linked with some libraries that force the use of a single
library when it initialize. We did not identify the "guilty" library, but
it seems to be linked with OpenCV.

Anyway, here is our temporary solution for those who want to make their own
filter with parallel processing (ThreadedGenerateData)

in the CMakeLists.txt, add the following line just after "project"

SET(CMAKE_EXE_LINKER_FLAGS "-Wl,--no-undefined -Wl,--as-needed" CACHE
INTERNAL "" FORCE)

it reduces the chances to load the "guilty" library as it only loads the
one that are usefull. I guess that this is a good practice and many of you
already do it, but we didn't.

Post by Julien R
Hi Victor,
Thanks for your answer. I've tried it but my problem is that it does not
fully use the threads (8 thread -> 12.5% on each thread, and I would like
100% for each thread)

Rashad

2018-10-05 12:54:43 UTC

Permalink

Post by Julien Radoux
With the help of my colleague, we found a workaround to the problem. It
seems to be linked with some libraries that force the use of a single
library when it initialize. We did not identify the "guilty" library, but
it seems to be linked with OpenCV.
Anyway, here is our temporary solution for those who want to make their
own filter with parallel processing (ThreadedGenerateData)
in the CMakeLists.txt, add the following line just after "project"
SET(CMAKE_EXE_LINKER_FLAGS "-Wl,--no-undefined -Wl,--as-needed" CACHE
INTERNAL "" FORCE)
it reduces the chances to load the "guilty" library as it only loads the
one that are usefull. I guess that this is a good practice and many of you
already do it, but we didn't.

see this doc
https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Modules/IO/TestKernel/test/CMakeLists.txt#L36

Post by Julien Radoux

Post by Julien R
Hi Victor,
Thanks for your answer. I've tried it but my problem is that it does not
fully use the threads (8 thread -> 12.5% on each thread, and I would like
100% for each thread)

Julien Radoux

2018-10-10 06:17:11 UTC

Permalink

Thanks for the link Rashad,

If I understand well, there is at least one ITK header that will not be
loaded with -Wl,-no-needed. Do you foresee other problems ? Note that it
runs for me with and without the option, but my problem is that it only
uses the equivalent of a single thread if I don't use the needed ones only.