Nrthugu

Previously I had asked a question regarding how to terminate thread blocked for I/O. I have used pthread_kill() instead of pthread_cancel() or writing to pipes, considering few advantages.

I have implementing the code to send signal (SIGUSR2) to the target thread using pthread_kill(). Below is the skeleton code for this. Most of the times getTimeRemainedForNextEvent() returns a value that blocks poll() for several hours. Because of this large timeout value, even if Thread2 sets terminateFlag (to stop Thread1), Thread2 gets blocked till poll() of Thread1 returns (which might be after several hours if there are no events on sockets). So I'm sending signal to Thread1 using pthread_kill() to interrupt poll() system call (if it gets blocked).

static void signalHandler(int signum) {

    //Does nothing

}



// Thread 1 (Does I/O operations and handles scheduler events). 



void* Thread1(void* args) {

    terminateFlag = 0;

    while(!terminateFlag) {

        int millis = getTimeRemainedForNextEvent(); //calculate maximum number of milliseconds poll() can block.



        int ret = poll(fds,numOfFDs,millis);

        if(ret > 0) {

            //handle socket events.

        } else if (ret < 0) {

            if(errno == EINTR)

                perror("Poll Error");

            break;

        }



        handleEvent();  

    }

}



// Thread 2 (Terminates Thread 1 when Thread 1 needs to be terminated)



void* Thread2(void* args) {

    while(1) {



    /* Do other stuff */



    if(terminateThread1) {

            terminateFlag = 1;

            pthread_kill(ftid,SIGUSR2); //ftid is pthread_t variable of Thread1

            pthread_join( ftid, NULL );

        }

    }



    /* Do other stuff */

} 

Above code works fine if Thread2 sets terminateFlag and sends signal to Thread1 when it blocked in poll() system call. But, If context switch happens after getTimeRemainedForNextEvent() function of Thread1 and Thread2 sets terminateFlag and sends signal, poll() of Thread1 gets blocked for several hours as it lost the signal that interrupts the system call.

It seems I can not use mutex for synchronization as poll() will hold the lock till it gets unblocked. Is there any synchronization mechanism that I can apply to avoid the above mentioned issue ?

In the first place, access to shared variable terminateFlag by multiple threads must be protected by a mutex or similar synchronization mechanism, else your program does not conform and all bets are off. That might, for instance, look like this:

void *Thread1(void *args) {

    pthread_mutex_lock(&a_mutex);

    terminateFlag = 0;

    while(!terminateFlag) {

        pthread_mutex_unlock(&a_mutex);



        // ...



        pthread_mutex_lock(&a_mutex);

    }

    pthread_mutex_unlock(&a_mutex);

}



void* Thread2(void* args) {

    // ...



    if (terminateThread1) {

        pthread_mutex_lock(&a_mutex);

        terminateFlag = 1;

        pthread_mutex_unlock(&a_mutex);

        pthread_kill(ftid,SIGUSR2); //ftid is pthread_t variable of Thread1

        pthread_join( ftid, NULL );

    }



    // ...

} 

But that does not solve the main problem, that a signal sent by thread 2 may be delivered to thread 1 after it tests terminateFlag but before it calls poll(), though it does narrow the window in which that could happen.

The cleanest solution is that suggested already by @PaulSanders' answer: have thread 2 wake thread 1 via a file descriptor that thread 1 is polling (i.e. by means of a pipe). Inasmuch as you seem to have a plausible reason to seek an alternative approach, however, it should also be possible to make your signaling approach work by appropriate use of signal masking. Expanding on @Shawn's comment, here's how it would work:

1. The parent thread blocks SIGUSR2 before starting thread 1, so that the latter, which inherits its signal mask from its parent, starts with that signal blocked.




2. Thread 1 uses ppoll() instead of poll(), so as to be able to specify that SIGUSR2 will be unblocked for the duration of that call. ppoll() does signal mask handling atomically, so that there is no opportunity for a signal to be lost when it is blocked before the call and unblocked within.




3. Thread 2 uses pthread_kill() to send SIGUSR2 to thread 1 to make it stop. Because that signal is only unblocked for that thread when it is performing a ppoll() call, it will not be lost (blocked signals remain pending until unblocked). This is precisely the kind of usage scenario for which ppoll() is designed.




4. You should even be able to do away with the terminateThread variable and associated synchronization, because you should be able to rely upon the signal being delivered during a ppoll() call and therefore causing the EINTR code path to be exercised. That path does not rely on terminateThread to make the thread stop.

Consider having an additional file descriptor in the set of fds passed to poll whose sole job is to make poll return when you want to terminate the thread.

Thus, in thread 2 we would have something like:

if (terminateThread1) {

        terminateFlag = 1;

        send (terminate_fd, " ", 1, 0);

        pthread_join (ftid, NULL);

    }

}

And terminate_fd would be in the set of fds passed to poll by thread 1.

-- OR --

If the overhead of having an extra fd per thread is too much (as discussed in the comments) then send something to one of the existing fds that thread 1 ignores. This will cause poll to return and then thread 1 will terminate. You can even have this 'special' value act as the terminate flag, which makes the logic a little tidier.

As you say yourself, you could use thread cancellation to solve this.
Outside of thread cancellation, I don't think there's a "right" way to solve this within POSIX (waking up the poll call with a write isn't exactly a generic method that would work for all situations in which a thread might get blocked), because POSIX's paradigm for making syscalls
and handling signals simply doesn't allow you to close the gap between a flag check and a potentially long blocking call.

void handler() { dont_enter_a_long_blocking_call_flg=1; }

int main()

{  //...

    if(dont_enter_a_long_blocking_call_flg)

        //THE GAP; what if the signal arrives here ?

        potentially_long_blocking_call();

    //....

}

The musl libc library uses signals for thread cancellation (because signals can break long-blocking calls that are in kernel mode)
and it uses them in conjunction with global assembly labels so that from the flag setting SIGCANCEL handler, it can do
(conceptually, I'm not pasting their actual code):

void sigcancel_handler(int Sig, siginfo_t *Info, void *Uctx)

{

    thread_local_cancellation_flag=1;

    if_interrupted_the_gap_move_Program_Counter_to_start_cancellation(Uctx);

}

Now if you changed if_interrupted_the_gap_move_Program_Counter_to_start_cancellation(Uctx);
to if_interrupted_the_gap_move_Program_Counter_to_make_the_syscall_fail(Uctx); and exported the if_interrupted_the_gap_move_Program_Counter_to_make_the_syscall_fail function along with the thread_local_cancellation_flag.

then you can use it to:

• solve your problem robustly
  implement robust signal cancelation with any signal without having to put any of that pthread_cleanup_{push,pop} stuff into your already working thread-safe singel threaded code


• ensure assured normal-context reaction to a signal delivery in your target thread even if the signal is caught.

Basically without a libc extension like this, if you once kill()/pthread_kill() a process/thread with a signal it handles or if put a function on a signal-sending timer, you cannot be sure of an assured reaction to the signal delivery, as the target may well receive the signal in a gap like above and hang indefinitely instead of responding to it.

I've implemented such a libc extension on top of musl libc and published it now https://github.com/pskocik/musl. The SIGNAL_EXAMPLES directory also shows some kill(), pthread_kill, and setitimer() examples that under a demonstrated race condition hang with classical libcs but don't wit my extended musl. You can use that extended musl to solve your problem cleanly and I also use it in my personal project to do robust thread cancellation without having to litter my code with pthread_cleanup_{push,pop}.

The obvious downside of this approach is that it's unportable and I only have it implemented for x86_64 musl. I've published it today in the hope that somebody (Cygwin, MacOSX?) copies it, because I think it's the right way to do cancellation in C.

In C++ and with glibc, you could utilize the fact that glibc uses exceptions to implement thread cancellation and simply use pthread_cancel (which uses a signal (SIGCANCEL) underneath) but catch it instead of letting it kill the thread.