Add reactor option to start with monotonic clock (was: LoopingCall problem)

[rfg007]

I wrote a simple test trying to find why my app freeze when change the system time:

from twisted.internet import reactor, task
import time


def callback():
    print time.ctime(time.time())




t = task.LoopingCall(callback)
t.start(2.0)
reactor.run()

If run this and change the system time, let's say a day forward, everything work fine, but if change the system time backward, the program freeze until you change forward again. This is specially problematic for a computer using ntp, updating its clock time to time.

It seems that there is a problem in LoopingCall._reschedule that don't play well when "fromNow = self.starttime - self._seconds()" is negative, and seems to be platform independent:

W2k Pro python 2.4.4 twisted 2.4.0

Wxp Pro python 2.4.4 twisted 2.4.0

Ubuntu Linux 5.1 python 2.4.2 twisted 2.0.1-4

[exarkun]

That loop could definitely be improved. It's really, really hard to fix this completely, though. The problem is that callLater's behavior is affected by changes in the system time. If callLater(2, f) happens, and then one second later the system clock is set back five seconds, six more seconds must pass before f will be called. There may also be a problem with the line you pointed out, but even if that is fixed, this callLater behavior will remain.

Fixing this may require using something like the POSIX monotonic clock. However, accessing this from Python is difficult, integrating it with select(2) is non-obvious, and I'm not sure what equivalents exist on non-POSIX platforms.

[rfg007]

Thanks Glyph,

I was looking in the code what you say, and I understand the problem. I saw the implementation of callLater and DelayedCall. Maybe if I introduce a simplification to my problem, you can guide me in a workaround.

Supose that the system time don't change automatically and, my application knows when to change the system time, and can change it. That is, it knows the actual system time and the required one.

Q: can be all the pending tasks re-scheduled according to the remaining time to complete them, after the app change the time?

PD: for simplicity, I omit in the OP that I use taskLoopingCall and tksuport in the reactor, and tk use its internal timers .after(xx_ms, callback) for other updates in the gui.

[therve]

#3899 has been closed as a duplicate of that one. I'm wondering if #1396 wouldn't solve this as well.

[ivank]

If Twisted eventually implements a monotonic clock to fix this, it would be a breaking change, because it would cause DelayedCall.getTime to sometimes return the monotonic clock time. Some programs assume dc.getTime() returns something close to the wall-clock time. This includes Twisted (see #4198) and Foolscap (as of r608:836b3aececec), in foolscap/reconnector.py. I assume other third-party programs would be affected as well. As glyph mentioned in #4198, the correct thing for DelayedCall users is (probably) to use dc.seconds() instead of time.time() when doing arithmetic with dc.getTime() results.

One solution is to add some flag to enable the monotonic clock, but I don't think most people will find the flag before they suffer a DelayedCall issue when their clock jumps.

As a side note, libev has some interesting code that detects clock jumps and adjusts delayed calls, even without access to a monotonic clock.

[exarkun]

Making this a reactor option has all sorts of problems:

• How do you specify it (most programs don't accept any arguments for reactor configuration; twistd and trial stand out in particular here; there's also a plugin API for reactors which doesn't accept any configuration)?
• What if application code wants to follow the system clock?
• What if two different pieces of application code in one process want different things?

Rather, introducing a new API with the new desired behavior makes more sense, I think.

[glyph]

Replying to exarkun:

Making this a reactor option has all sorts of problems: * How do you specify it (most programs don't accept any arguments for reactor configuration; twistd and trial stand out in particular here; there's also a plugin API for reactors which doesn't accept any configuration)? * What if application code wants to follow the system clock? * What if two different pieces of application code in one process want different things? Rather, introducing a new API with the new desired behavior makes more sense, I think.

+1.

Monotonic clocks are super useful, but they are something really distinct from callLater. Granted, many uses of callLater would probably prefer a monotonic clock, but we should leave it up to them to decide that explicitly.

[ivank]

I will note that nearly all existing code is using callLater for timeouts and equally-spaced actions (which should be scheduled by a monotonic clock). My perspective is that not using the monotonic clock by default is a bug, but unfortunately that is not backwards-compatible.

1. I guess the two options are: (1) add an argument to every program that eventually calls reactor.run, or (2) check os.environTWISTED_ALLOW_MONOTONIC?

2. If any application code wants to use callLater to follow the system clock, the developer must not enable "reactor with monotonic clock". In the future there could be some new API to follow the system clock that works even when "reactor with monotonic clock" is enabled.

3. Newly-written code should support both normal "reactor" and "reactor with monotonic clock". If any applications a developer wants to run in their process do not work with "reactor with monotonic clock", the developer must not enable the option.

[ivank]

I am attaching a patch that makes the reactor use a monotonic clock when available, with no configuration options. It is useful for checking which applications break if the reactor uses a monotonic clock. The patch also includes a fix for #4198.

The patch requires Monoclock from  http://ludios.net/git/ , which probably only works on Linux.

[itamar]

I propose:

1. Use ctypes to implement this, so it works on pypy.
2. Monotonic time is always on.
3. DelayedCall.getTime makes a best effort to figure out a wallclock time... but is deprecated.
4. The replacement is a method called getRemainingTime that returns a relative time, and it's documented to be relative to reactor.seconds only.

[zooko]

comment:13 all sounds good to me!

Here is something which might be useful to an implementor of this.

The new C++ standard defines something called steady_clock which is even better than a monotonic clock. A monotonic clock is one that can never be adjusted backward to an earlier setting. A steady clock is one that can never be adjusted.

 http://en.cppreference.com/w/cpp/chrono

The reason this might be useful to an implementor of this ticket is that you might be able to crib some bits from an implementation, such as that of the Boost library.

Here is the header file:

 http://svn.boost.org/svn/boost/trunk/boost/chrono/config.hpp

This shows that Boost implements steady_clock on Windows, Mac OS X, and "sun", and on POSIX if any of: * CLOCK_REALTIME and CLOCK_MONOTONIC are defined, or * _POSIX_THREAD_CPUTIME, or * CLOCK_THREAD_CPUTIME_ID

The implementations are in boost/chrono/detail/inlined.

Here are the cores of the implementations. (I've trimmed off some macros, namespaces, error-checking, and optimizations and haven't tested these. Best use the original Boost code for any actual use -- the following is just for illustration.):

*  the windows implementation

  double get_nanosecs_per_tic()
  {
      boost::detail::win32::LARGE_INTEGER_ freq;
      boost::detail::win32::QueryPerformanceFrequency( &freq );
      return double(1000000000.0L / freq.QuadPart);
  }

  steady_clock::time_point steady_clock::now()
  {
    static double nanosecs_per_tic = get_nanosecs_per_tic();

    boost::detail::win32::LARGE_INTEGER_ pcount;
    boost::detail::win32::QueryPerformanceCounter( &pcount );

    return steady_clock::time_point(steady_clock::duration(
      static_cast<steady_clock::rep>((nanosecs_per_tic) * pcount.QuadPart)));
  }

*  the mac implementation

double
compute_steady_factor()
{
    mach_timebase_info_data_t MachInfo;
    mach_timebase_info(&MachInfo);
    return static_cast<double>(MachInfo.numer) / MachInfo.denom;
}

steady_clock::rep
steady_full()
{
    static const double factor = chrono_detail::compute_steady_factor();
    return static_cast<steady_clock::rep>(mach_absolute_time() * factor);
}

*  the posix implementation

  steady_clock::time_point steady_clock::now()
  {
    timespec ts;
    ::clock_gettime( CLOCK_MONOTONIC, &ts );
    return time_point(duration(
      static_cast<steady_clock::rep>( ts.tv_sec ) * 1000000000 + ts.tv_nsec));
  }

(Hm, I don't see the implementation on posix when CLOCK_MONOTONIC *isn't* available...)

[exarkun]

Use ctypes to implement this, so it works on pypy.

Using ctypes may just guarantee that it doesn't reliably work anywhere, not that it reliably works on CPython and PyPy. ;) For example, there's no way to find the value of CLOCK_MONOTONIC using ctypes. This is not to say that I think we should have an implementation of this in C, just to point out that ctypes has serious drawbacks as well.

Monotonic time is always on.

I'm not sure what this means. Perhaps it means that we change the behavior of reactor.callLater to mean "call this after N seconds", rather than "call this at wallclock time now plus N seconds"? Why would be do that, rather than introduce a second scheduling API? Phrase your answer without contracting CompatibilityPolicy.

DelayedCall.getTime makes a best effort to figure out a wallclock time... but is deprecated.

This could be okay, perhaps. On the other hand, why deprecate wall clock time support? Can it be implemented without help from the reactor?

The replacement is a method called getRemainingTime that returns a relative time, and it's documented to be relative to reactor.seconds only.

Yay, new APIs. That's more like it. This implies that reactor.seconds changes its meaning as well, though. Perhaps it should be left alone and a new monotonic time source should be introduced instead?

A separate point which I don't see addressed anywhere on this ticket, yet, is how CLOCK_MONOTONIC interacts with the timeout parameter to select, epoll_wait, kevent, etc. Perhaps this is obvious to everyone else and there's no special consideration required in the implementation; if so, it would be nice for someone to say so. :)

[itamar]

I am unhappy with a new API because it means everyone has to change their existing code. OTOH a pluggable time policy (or changed default) might in theory break existing code, yes.

As far as epoll_wait and friends, it looks like one possibility is to use a timerfd with CLOCK_MONOTONIC (though that's Linux only). Moreover libevent has monotonic time support with epoll et al so probably we can look at their implementation as an example. And quite possibly their support is cross-platform so educational in general.

[zooko]

Replying to exarkun:

Monotonic time is always on.

I'm not sure what this means. Perhaps it means that we change the behavior of reactor.callLater to mean "call this after N seconds", rather than "call this at wallclock time now plus N seconds"? Why would be do that, rather than introduce a second scheduling API? Phrase your answer without contracting CompatibilityPolicy.

What effects would this have on application? The effect that I am concerned about is "what happens when the system clock gets adjusted?". If you could explain what other effects there could be (possibly related to the aforementioned DelayedCall.getTime?), that would help me think about this.

As for the "what happens when the system clock gets adjusted?" issue, there are three possible classes of application:

• Those that rely on their events executing N seconds later and in the expected order, regardless of whether the system clock gets adjusted. • Those that rely on the timing or ordering of their events getting changed when the system clock gets adjusted. • Those that don't care when or in what order their events occur.

My intuition is that all or almost all applications fall into the first category, and changing the behavior of this API would be fixing a bug for them rather than introducing an incompatibility. For applications that fall into the second category, changing this would be incompatibly changing the API.

Apparently other people have a different intuition about this. Could you help me understand by providing examples of actual or plausible applications of the second category -- those that need their events to change timing or ordering because their system clock got reset?

[exarkun]

I am unhappy with a new API because it means everyone has to change their existing code.

Not everybody. Just applications that want the new behavior and not the old behavior - ie, applications that were broken before and want to be fixed. Specifically, applications that were correct before require no change.

If possible, I would prefer to not have to fight about whether to preserve backwards compatibility or not. That is why we have the CompatibilityPolicy, is it not? If you think the change is "compatible enough" (whatever that might mean), then I think we need to find some way to resolve this kind of disagreement.

[glyph]

Replying to exarkun:

A separate point which I don't see addressed anywhere on this ticket, yet, is how CLOCK_MONOTONIC interacts with the timeout parameter to select, epoll_wait, kevent, etc. Perhaps this is obvious to everyone else and there's no special consideration required in the implementation; if so, it would be nice for someone to say so. :)

Some time ago, when investigating a related issue, I did a manual test of the behavior of select's timeout on various platforms. As I recall, it generally behaves as though it is using a monotonic timer; of course, these were very coarse-grained tests and I was just verifying them by eye. I assume that some kind of run-as-root script that knew about both monotonic and wall clocks could be constructed to produce useful output.

Sometimes, according to rules I don't really understand, setting the system clock would instead cause a waiting select to time out immediately; a strict reading of  the relevant standard seems to say that it should not do this, but then, it says absolutely nothing about changes to the system clock. However, I've not been able to reproduce this behavior in my more recent testing, and maybe it was a bug in some platform at the time (circa 3 years ago) which has since been fixed.

[zooko]

Oh, I just noticed this:

 http://docs.python.org/dev/library/time.html#time.monotonic

It looks like it will be in CPython 3.3 and uses the posix CLOCK_MONOTONIC_RAW if present, else the CLOCK_MONOTONIC. On Windows it uses QueryPerformanceCounter.

[zooko]

This README from monoclock has an interesting TODO, to wit:  https://github.com/ludios/Monoclock/blob/master/README.md

Support buggy AMD chips, or expose a probablyBuggy() function that returns True if the monotonic clock is unreliable.

Note: Chromium's base/time_win.cc just disables use of the monotonic clock on Athlon X2 CPUs with if (cpu.vendor_name() == "AuthenticAMD" && cpu.family() == 15

[zooko]

Here's a ctypes implementation:  http://code.google.com/p/anacrolix/source/browse/projects/pimu/monotime.py

[zooko]

I told JP in person at Pycon about how I wrote my own workaround to protect my programs from system clock unsteadiness years ago. (I discovered the issue because the system clock was jumping around and causing my events to fire in a different order than I had told them to!) Here's the evidence of that:  https://tahoe-lafs.org/trac/pyutil/browser/trunk/pyutil/increasing_timer.py?annotate=blame . The in-line comments mention it being 2001. I told him how when I switched from my own homegrown framework to Twisted I abandoned this measure because I trusted Twisted to (sooner or later) handle it for me.

I said that if there were an API named reactor.callAtDateAndTime, that the programmer would expect the event to go off at the nominal date, on the *new* system clock, if the system clock changed after the call to reactor.callAtDateAndTime but before the event happened. On the other hand, since the API is reactor.callLater and the first argument is named delay, the programmer expects the event to happen delay seconds afater the call to reactor.callLater, regardless of whether the system clock changes in the interim.

JP said he didn't object to this change in principle, but he wanted someone to test it. Manually, that is, since it is awfully inconvenient to automatically, virtualizably, reproducibly test the behavior of your OS kernel...

To test this, I think you would write a Python script which starts two threads. One sleeps for a second while the other calls select with a given timeout, then the first wakes up and sets the system time (maybe by running sudo date --set in a subprocess). Then the human who is sitting there watching with his wristwatch in hand sees whether the duration of the call to select is affected by the change of clock. exarkun: is that what you meant?

There is an open issue about whether to use CLOCK_MONOTONIC or CLOCK_MONOTONIC_RAW on platforms that offer both (which means: on Linux). I don't mind that much either way. I think I would prefer CLOCK_MONOTONIC, which if I understand correctly promises that even though it will never ever cause the clock to jump, either forward or backwards, it *will* cause the clock to run a bit faster or slower in the attempt to keep the clock's rate close to true. One hesitation I have about that is I don't know if some implementations of CLOCK_MONOTONIC might try to make the clock run faster in order to "catch up" to another clock, which to my mind violates the principle of the thing. On the other hand the drawback of CLOCK_MONOTONIC_RAW is that the clock will be chronically a little fast or a little slow (how much, I don't know. An experiment is probably in order), and may speed up when the clock is hotter and slow down when it is colder. That isn't ideal behavior, but at least is more likely that its behavior will be independent of bugs, overly helpful Linux kernel hackers, and potentially malicious ntp servers.

I'll bet there are very few people who know if the behavior of the "monotonic" clock on non-Linux platforms is more like linux's CLOCK_MONOTONIC or more like linux's CLOCK_MONOTONIC_RAW... I am not among them.

By the way the Boost and C++ folks have decided that the right name for this is a "steady clock" instead of a "monotonic clock". After all, the word "monotonic" means "it goes only in one direction", but leaves open the question in the mind of the programmer about whether it might jump *forward* in that direction in the attempt to catch up with someone else's standard of time. The word "steady" better conveys the truth: that it never jumps in either direction.

[exarkun]

#5552 was a duplicate of this.

[glyph]

I had a look at #5552 and the patch attached here and realized that neither of them dealt with OS X and one required an additional C extension.

I will attach a version that provides a monotonic clock for both Linux and OS X.

[glyph]

monotonic clock for linux & OS X

[glyph]

Another thing to keep in mind as someone implements this is that system sleep will cause pretty much every platform's monotonic clock to leap forward; it doesn't increment while the system is in S3 or higher. Unfortunately it looks like the suggested way to work around this is to simply look at the wall clock and trust that it's right! So I started looking into what notifications various platforms can give you.

First up; linux. It seems like time on linux is a real mess. The kernel won't tell you anything about power management unless you're a driver. Luckily, you can mostly ignore suspend on servers (at least for now) since they don't sleep. Laptops, where this is more of a concern, are likely going to be running some kind of desktop environment, where you have access to the pleasant  Sleeping and  Resuming D-Bus signals. However, there's no particular obligation on linux to use UPower to manage your power, so suspend might happen through some other mechanism and you won't be told about it. Too-clever-by-half users who like to cat things into /proc directly will break these notifications. Right now there's no way to get told about settimeofday, but if we're lucky,  in the far-flung future you might be able to call sys_time_change_notify. (FreeBSD seems to be in basically the same boat, without the hope for the future.)

Next: Windows. As usual, everything is kind of gross, but possible. The messages for  WM_POWERBROADCAST and  WM_TIMECHANGE should allow a process to figure out when power and time changes occur. I'm not sure how much ctypes grossness this entails or how exactly this integrates with IOCP and win32reactor.

On OS X, everything happens through NSNotificationCenter, and assuming we're using CFReactor, I believe we can fairly easily receive  NSWorkspaceWillSleepNotification, NSWorkspaceDidWakeNotification, and  NSSystemClockDidChangeNotification in python.

I don't think this information has ever been brought together in one place before, so hopefully this will serve as a useful reference to the future implementor of this feature for Twisted.

[zooko]

If I call reactor.callLater(10, do_something), and one second later the laptop is suspended, and 24 hours later the laptop is unsuspended, do I expect for do_something to happen ASAP, or to wait 9 more seconds after unsuspend?

In other words, was the "10" to mean that 10 seconds have passed in the real world which the computer inhabits, or 10 seconds of awake computer time have passed?

I think maybe each alternative is sometimes intended by programmers, which means Twisted can't do a generically correct thing here unless we extend the API so the programmer can express what they want in this case.

[glyph]

Replying to zooko:

If I call reactor.callLater(10, do_something), and one second later the laptop is suspended, and 24 hours later the laptop is unsuspended, do I expect for do_something to happen ASAP, or to wait 9 more seconds after unsuspend? In other words, was the "10" to mean that 10 seconds have passed in the real world which the computer inhabits, or 10 seconds of awake computer time have passed? I think maybe each alternative is sometimes intended by programmers, which means Twisted can't do a generically correct thing here unless we extend the API so the programmer can express what they want in this case.

Quite so.

If you call sleep(10); do_something(); kernel folks seem to have generally taken the opinion that that means "10 seconds of awake computer time". There are some mailing list threads (which I can't find right now) where at least Linux kernel people have publicly said that this behavior is quite intentional, as if it were interpreted the other way, every laptop unsuspend would have every process waiting on a timeout (which is apparently a lot of them) getting scheduled to run instantly and consuming all your power. Following that lead, I think that maybe callLater should mean "computer awake time" and we should add something higher-level like callAtTime which would take advantage of those power-management and clock-adjustment notifications I mentioned in my earlier comment.

[zooko]

+1

I read some of the linux kernel source in kernel/time/timekeeping.c, and I saw that it makes sure CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW show "computer awake time", but it offers a different function to get a steady measurement of time from the perspective of someone outside the computer.

[itamar]

As far as backwards compat, it seems like DelayedCall.getTime and reactor.seconds are the main issues, in case someone uses them in ways that interact with some other source of time (e.g. time.time). I would suggest returning a new object that:

1. Includes a type info about what kind of time it is (clock vs. monotonic vs. fake for testing).
2. When added/compared/etc. to floats or ints does a DeprecationWarning.

[zooko]

Itamar: that sounds like a good idea to me!