[Twisted-Python] buildbot.twistedmatrix.com is down a lot

[Adi Roiban]

On 20 July 2016 at 17:51, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:

>
> On Jul 20, 2016, at 6:31 AM, Adi Roiban <adi at roiban.ro> wrote:
>
>
>
> On 18 July 2016 at 19:04, James Broadhead <jamesbroadhead at gmail.com>
> wrote:
>
>> On 17 July 2016 at 07:21, Amber Brown <hawkowl at atleastfornow.net> wrote:
>>
>>> It's OOMing  (...)
>>>
>>
>> Have you considered something like monit[1] to detect & restart in cases
>> like this?
>>
>>
> This might help, but will not help up understand what we are doing wrong :)
>
> After disabling the github webhooks, the buildbot look stable... so we
> might have a clue about what goes wrong.
>
> Right now I don't have time to look into this issue, so github hooks are
> disabled for now from the GitHub UI.
>
>
> Can someone who's had a direct look at the OOMing process (adi? amber?)
> report this upstream?  It's a real pity that we won't get github statuses
> for buildbot builds any more; that was a huge step in the right direction.
>
>
I don't know how to grasp this.
By the time I was observing the issue, the buildbot process was already
dead.

I have recently discovered the Rackspace monitoring capabilities for VM...
and set up a memory notification... not sure who will receive the alerts.

I have re-enable to GitHub hooks and will start taking a closer look at the
buildmaster process.... but maybe 2GB is just not enough for a buildmaster.

I have triggered the creation of an image for the current buildbot machine
and will consider upgrading the buildbot to 4GB of memory to see if we
still hit the ceiling.

For my project I have a similar buildmaster based on number of builders and
slaves (without github hooks and without linter factories) and in 2 weeks
of uptime the virtual memory usage is 1.5GB
.... so mabybe 2GB is just not enough for buildbot.

-- 
Adi Roiban
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20160720/efb76a2a/attachment-0002.html>