path to restoring FreeBSD to Tier 2 support

[Trott]

What would have to happen to restore FreeBSD to Tier 2 support? Here's the tiniest of beginnings of a task list. Feel free to add to it. @nodejs/build

• Need a volunteer to do sys admin, troubleshooting, documentation, ansible-scripting, etc. for FreeBSD in CI. (If that person is an employee of a company invested in Node.js/FreeBSD, that would be a big plus, but not strictly a requirement.)

• ... what else?

[Trott]

Refs: nodejs/node#27383

[nschonni]

There is a public FreeBSD CI service now too https://forums.freebsd.org/threads/cirrus-ci-support-for-freebsd.68644/

[Trott]

Do we need to acquire one or more people to be the go-to FreeBSD experts when it comes time to create a release? I imagine that would be the Build WG member or members. I assume we don't need someone on the Release WG for that. But typing this out loud in case I'm wrong.

[Trott]

There is a public FreeBSD CI service now too https://forums.freebsd.org/threads/cirrus-ci-support-for-freebsd.68644/

Thanks! I do think we're going to continue to need FreeBSD in our existing CI. Public CI is not an option in a lot of situations for us, such as when we are doing security releases. And even if it were, I'm not sure fragmenting our CI across multiple services is a path to less maintenance pain instead of more (but I could be wrong!).

[Trott]

Putting on the meeting agenda even though I only opened it 15 minutes ago because:

• I believe this is worth talking about very publicly.
• This does not seem like the type of thing that warrants weeks of open-ended discussion. I believe if we had a 10-minute synchronous conversation, we could probably have the checklist completed. So let's just do that.

[refack]

IIUC the idea behind the downgrade is to "officially-unofficially" move freeBSD to the community-contrib path (#1788. Similar to Linux/ia32 and ARM6)

[refack]

• Need a volunteer to do sys admin...

I believe the other part is maintaining a robust @nodejs/platform-freebsd team to own the code side...

[targos]

• Need a volunteer to do sys admin...

I believe the other part is maintaining a robust @nodejs/platform-freebsd team to own the code side...

Yes! I think this comment explains a bit what our problem is regarding FreeBSD support: nodejs/node#27383 (comment)

If FreeBSD is a release-blocking platform, we have to fix any build issue in nodejs/node. It happened several times that a V8 upgrade was blocked only because of FreeBSD and we spent more time than necessary to fix the issues because no expert was here to help. Since issues had to be fixed by us in the first place, it might be why the FreeBSD port "just works".

[nomadlogic]

If FreeBSD is a release-blocking platform, we have to fix any build issue in nodejs/node. It happened several times that a V8 upgrade was blocked only because of FreeBSD and we spent more time than necessary to fix the issues because no expert was here to help. Since issues had to be fixed by us in the first place, it might be why the FreeBSD port "just works".

it sounds like a lot of this is due to lack of communication between teams. aside from being sure FreeBSD developers are subscribed to this and the main nodejs issue feeds, are there other less noisy ways to ensure problems like you mention do not get missed? personally speaking i'm sub'd to nodejs/build and nodejs/node projects on github (with some filtering to lower the signal/noise ratio), but are there other communication channels FreeBSD should also be keeping an eye on?

[refack]

but are there other communication channels FreeBSD should also be keeping an eye on?

We have the @nodejs/platform-freebsd team, but for $GITHUB_REASONS you need to be a members of the Org to subscribe to that team's notifications.

[sam-github]

I'd personally be OK with adding FreeBSD members as nodejs collaborators even if they don't have a nodejs contributions history, if that's what it takes for them to be get notified of issues, and start being contributors.

[rvagg]

On the infra side: it would be good to have some FreeBSD natives who are somewhat present around nodejs/build (casually following the repo and watching for ways to jump in) and in the nodejs/platform-freebsd team as well so we can use that to summon help where needed.

But we still have a problem with turn-around, @Trott this goes back to our discussion about urgency and downtime: there's one class of problem where there's just something that needs to be taken care of but is not super urgent (like FreeBSD machines not having an auto-restart mechanism for the Jenkins process which was a huge headache for the longest time until someone finally put monit on them). But then there's another class of problem where something's failing right now and someone has to fix it. That falls on those of us who are more present—who watch the repo, who are on IRC and who tend to get pinged when there are urgent matters (because we tend to be responsive: myself, Refael and Michael get that the most at the moment I think).

This second class is where the pain is felt and I don't think that can be solved by casual involvement. The reason FreeBSD was easy for us earlier on was because we had Johan who was a core Build member, was very active and available, he was also a FreeBSD native (which is why we still have things like the linter running on FreeBSD). Without a Johan around these urgent things fall on me and the other active members and this is where the biggest problem is.

Back to my comments on nodejs/node#27383: the experience for me of managing FreeBSD and picking up pieces when things fail (combined with a brief and disastrous run with FreeNAS at home), have made me really dislike FreeBSD. And it's my strong impression that its presence in our CI system is much more of negative than a positive for the work that we need to get done here. I feel almost exactly the same way about SmartOS to be honest but it gets more of a free pass because at least Joyent is a major infrastructure contributor (and Colin et. al. are usually responsive to calls via nodejs/platform-smartos).

Our infrastructure make-up is really driven very strongly by the individuals, and companies, that contribute here. It's not all about Node core needs, there's a big element of preferences. e.g. we have a ton of Ubuntu in our mix, it runs some our most important servers and it's the base OS for a lot of our Docker container work—that's almost entirely about my personal preferences, not necessarily technical merit. If Johan was still an active contributor then we'd probably see a much stronger presence of FreeBSD and Debian.

If you like FreeBSD, then there's a good chance you're a bit of a systems nerd and you might even enjoy the work we do around here across many systems. So maybe you should consider getting more involved and becoming one of those people that reacts to urgent needs. If we had that, and the core dev team didn't have the sense that FreeBSD gets in the way (as @targos pointed out above), then I'm sure it could easily raise in status.

[mhdawson]

As Rod said its all about having one or more people who are active enough in the project and build WG to champion FreeBSD and be responsive enough to make sure it is not a burden on others. This includes the full cycle from helping to keep the machines configured and up to date following the build WG standard practices (ansible templates etc.) to being responsive when there are machine issues or test failures on the platform.

[sam-github]

nodejs/node#23089

^---- Just bumped into this while cruising for flaky tests to fix. Perhaps its a relevant example? For FreeBSD to move up a tier there needs to be people who notice issues like this with "freebsd" in them, and respond to them.

[sam-github]

Does this need to be discussed in a WG meeting? It seems better discussed here.

[sam-github]

@emaste I've never been involved in decisions about Tier, so take this with a grain of salt. That said, I do work on IBM platforms, and am under no illusions about what Tier they would be if we didn't work very hard to make sure we are actively and noticably working on problems related to them.

Actively triaging anything with freebsd labels here and in node would be helpful:

https://github.com/nodejs/node/issues?q=is%3Aopen+is%3Aissue+label%3Afreebsd

I didn't hear back about any efforts to repro nodejs/node#23089

One of the things about FreeBSD is that for people who don't use FreeBSD, we can only troubleshoot on the CI systems, which are fairly locked down (for obvious reasons). The hope would be that FreeBSD devs have FreeBSD boxes in front of them, and can repro lots of issues immediately without needing perms on CI.

Reviewing any tests marked flaky on FreeBSD would help (the *.status files in test/...).

And again, this is not some final word (I don't have that power), but I suspect one thing that would help is that every time platform-freebsd was tagged, if someone popped there head up and said "I'll take care of this", that would probably get FreeBSD back to Tier 1. That's a bit fuzzy, but honestly, no one is keeping track of exact response times for FreeBSD pings... that alone is the kind of thing that hopefully people wanting FreeBSD back in Tier 1 would be able to do, and use as evidence that it has responsive suport.

As you notice, its not that it has tons of difficulties, its a unixlike platform, and node runs pretty well on it. Well, outside of Rod's complaints about supporting it in our CI infrastructure. For that, it would be handy to have a FreeBSD primary person on the build team. Not too sure how to manage that, but I don't think anybody has offered (my memory could be wrong on that).

[nomadlogic]

Hi Sam, not speaking for Ed or the FreeBSD project but as an individual developer whose primary platform is FreeBSD. I've posted an update to nodejs/node#23089.

I've also been keeping a closer eye on the lists for issues related to FreeBSD and from what I've observed is that in the couple times FreeBSD has been suspected to be the root cause of an issue I've been to help determine what the cause is and amplify the bug report to a wider audience. To be honest several of the issues that have cropped up are not FreeBSD issues (for example there is an issue with a python library that is used to convert unit-test results, its not python3 compatible which is why those errors where happening).

Aside from keeping an eye on the bug lists and checking tests marked at flaky is needed? Are there specific tasks needed to improve FreeBSD's state on the build clusters?

[sam-github]

I don't if there are problems with FreeBSD on the build clusters. You could look in nodejs/build/issues to see if any are specific to it. @rvagg or @mhdawson might have ideas.

It might be worth auditing the python state, I've been updating pythons on some systems. Start here: #1674

[emaste]

I chatted with @mhdawson in the scheduled call - in addition to triaging individual issues found by CI, it would be good if FreeBSD folks are the ones to undertake upgrades of the FreeBSD CI build hosts, and be the ones to discover side effects of default Python version changes etc.

And then even better, if folks can help with generally maintaining the CI system (not just FreeBSD issues).

[sam-github]

Also, you might find it useful to monitor the build histories of the freebsd ci machines, and check the causes of failures, and find flaky or unstable tests in CI on FreeBSD before someone else does.

[sam-github]

nodejs/node#28735 <--- I've no idea how to evaluate if its correct or not, maybe someone here can review? Thanks.

[Trott]

We would need a path to getting FreeBSD 12 into our Jenkins setup. Not sure if there's someone around who would want to do that work, but no way to know until you put the call out so here's a comment.

[rvagg]

I still have #1871 on my TODO list, getting swap increased on existing machines. Once I've done that I might spare some thought to FreeBSD 12 and see if any of our infra donors are offering it—not stopping anyone else who has access from doing this, just saying that if nobody else does it then I'll get to it.

[rvagg]

removed from meeting agenda, I don't think there's much we can do in meetings, we can be much more productive having ongoing dialog here.

[emaste]

Where are the current FreeBSD instances running?

[nomadlogic]

@emaste I think they are hosted on Digital Ocean, Joyent and RackSpace:

https://ci.nodejs.org/computer/
(search for freebsd)

[Trott]

FWIW, Joyent and Rackspace are FreeBSD 10 only, I think. FreeBSD 11 hosts (and one FreeBSD 10 host) are at Digital Ocean.

[anandsuresh]

If this issue is still open and there is a need for someone to assist with testing node.js on FreeBSD, I'd like to help. We (Voxer) are moving our infrastructure from Joyent to Google Cloud and consequently migrating from SmartOS to FreeBSD 12. We're using node.js v8 for our production stack and are building a new production stack on v10, which will likely be upgraded to v12 once it is promoted to LTS.

Is there a team/channel I could join to get in touch with the right person(s) to figure out how to proceed?

[sam-github]

@anandsuresh are you a collaborator? If so, I can add you to the freebsd platform team. If you aren't, look for the freebsd label in nodejs/node and nodejs/build issues, and look for any tests in nodejs/node that are skipped on freebsd. Fix a couple of those things and you'll be a collaborator pretty soon!

Also, read this thread top to bottom, nothing new has happened, any suggestions are likely as valid now as when they were made.

[targos]

@sam-github anyone can be in the platform-* teams. They don't give special permissions.

[Trott]

@anandsuresh I've sent you an invitation to join the platform-freebsd team. (I believe you won't be able to join unless you have two-factor authentication enabled on your GitHub account.)

[anandsuresh]

Testing on my local machine running FreeBSD 12.0-RELEASE-p10, I was able to build and test the latest code in master without any issues. The output from the test run is available at https://gist.github.com/anandsuresh/e8a769e19d157718115a4f9dd6c8de6f.

It seems that the issue is specific to FreeBSD v10 or v11. I don't have any resources/jails with those specs at the moment. Will try and set something up to test further.

[anandsuresh]

Ran the same test in a FreeBSD 11.3 jail and discovered that it failed with the following errors, but passed when I can the test command again. Is there a history of flaky tests in this suite?

=== release test-net-connect-local-error ===
Path: sequential/test-net-connect-local-error
assert.js:89
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ undefined
- 12347
    at onError (/wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/sequential/test-net-connect-local-error.js:26:10)
    at Socket.<anonymous> (/wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/sequential/test-net-connect-local-error.js:36:49)
    at Socket.<anonymous> (/wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/common/index.js:371:15)
    at Socket.emit (events.js:209:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: undefined,
  expected: 12347,
  operator: 'strictEqual'
}
Command: out/Release/node /wrkdirs/usr/ports/www/node/work/node-v12.10.0/test/sequential/test-net-connect-local-error.js
[02:57|% 100|+ 2742|-   1]: Done
gmake[1]: *** [Makefile:296: jstest] Error 1
gmake: *** [Makefile:330: test-only] Error 2

[anandsuresh]

If it helps, the machine I'm running is a 2011 Mac Mini with 16GB of RAM and 1TB of storage that has been wiped clean to run FreeBSD 12.0-RELEASE-p10 on the host, with 2 poudriere jails used to build the node.js port on FreeBSD 12.0-RELEASE-p10 and 11.3-RELEASE-p3.

[rvagg]

@anandsuresh at a guess it's IPv6 problems. On all our test machines we make sure that this line is in /etc/hosts, maybe try it:

::1 localhost.localdomain localhost

common.localhostIPv6 is ::1 so it needs to be present

[jdwx]

A couple of points, bearing in mind that I am far from an expert in these matters:

• The major problem may be related to FreeBSD 10, which has reached end-of-life. The version of clang that was shipped with FreeBSD 10 had (in my opinion) a number of problems. Those problems no longer exist on current release versions. Keeping the build going on currently-supported versions of FreeBSD may therefore be substantially easier than it was.

• If the issue is a lack of hardware resources (i.e. servers to build/test/CI FreeBSD 11 and/or FreeBSD 12), my company can donate VM's to meet that need. (I see above that FreeBSD 12 may be an issue with your current providers.) Who would I contact about that?

[rvagg]

FYI we now have no FreeBSD 10 running in CI, we just need to deprovision the VMs, #2012.

[mhdawson]

@jdwx it's just as much about having people who will tend/care for the machines, investigate machine issues and test failures. Would your company be able to donate some of your time to help on that front?

[jdwx]

@mhdawson:

"tend/care for the machines"

Yes.

"investigate machine issues"

Yes.

"test failures"

Probably not much. Our Node.JS experience is very limited and our collective knowledge of its internals amounts to basically nil.

The VM(s) we provide would be fully managed for you, but on the development side the best we could probably do is act as a resource for questions like, "WTF is FreeBSD actually doing here?" to the best of our ability; our knowledge of FreeBSD internals is substantial, but by no means exhaustive. But it sounds like maybe that's right about where the current gap is. So while it might not be as good as having all the relevant knowledge in one person's head, perhaps it would suffice?

[byalexandrepedrosa]

Just for add on discuss, I will share the history about my experience as consultant:

1 year ago I have left some FreeBSD servers from my clients because they was willing test the node.js and Vue / Nuxt (last one because native SSR ready) or other Javascript frameworks.

When they asked me to enable the support on FreeBSD 10 (currently LTS on that time), it was full of problems, and then clients was claiming about hire other consultants (so much time trying fixing errors) to do the Job, in another words, they does not care about if node.js was runing on Linux / FreeBSD on first time for they purpose / test ...

Well, for do not lose my contracts, then I have used Ubuntu and archive they requests...

After sometime, the number of OSs about fix stuff on Linux increased, and since I charge per call, they quit about use the JavaScript frameworks claiming those as headache…

Usually (making a big generalism) there 2 type of FreeBSD clients:

1 - Those with simple requirements focusing on stability, and since they requirements are simple the default ports works pretty well

2 - Some with advanced requirements trying get the stability of FreeBSD for enhance they heterogeneous (using FreeBSD everywhere is possible, and where is not, they use another best alternative) environment.

1 and 2 know about how solid FreeBSD is, and this is what make all difference… Is not bias or personal preference...

For those thinking this way (as personal opinion) so I suggest you to start pay for every call for your outsource company where system's problems occurs with popular Linux versions (Enterprise like Red hat since you pay an fee you can use they support and is not the point here).

Is that true mostly FreeBSD clients at some point don't renew they contracts for phew time, but when they come back, mostly of time the outdate environment still solid as rock (Got 2 months ago an request for update an FreeBSD 8 mail server to 12 for example), in another words, this is the casual client and is backing saying about "long term contract" and blah blah blah, but the truth, only wish upgrade they environment...

From point of view on enterprises, systems requiring less (read spend less money) support calls are the best option ever...
Nowadays I am testing again FreeBSD for node and Nuxt, this time FreeBSD 12.0 compiles everything on first try.

By the way, using PM2 for keep restarting the process have a lot of issues, seems they are looking how to fix the needed about run as root (never will use it on real production environment with this kind of behavior – if apache I Jail it and never run as root independent of they reputation, why would let pm2 or any other JavaScript running as root changing the entire environment…)

Second problem, speaking strictly about Javascript, imho there’s a big problem about keep everything update in sync…
Get the previous example…

On the PM2 github, mostly of “workarounds” about the previous issue is about downgrade node.js for specific version where “all was working”…
This kind of comment is the first factor to drop JavaScript from projects on meetings in IT enterprises, no one wish become the responsible for start projects with outdate versions with lot of CVE’s (security issues) only for speed up the development because one or another framework…
I will not discuss the problem about the repositories… Need be crazy for run arbitrary scripts on production servers as root, this is why the requirement about jail or use unpowered per process is the basic concept as security point of view…

Remembering running inside jail as root can ruin all your job too… Or in another words, if you will spend mostly time fixing things instead be productive, this will never be an good option.

I hope all discuss here by other people helping make node.js run pretty well on FreeBSD can consider those points, the idea is let you guys think about from enterprise perspective. In another words, for increase the popularity about node.js, the main way is support all systems possible, because every enterprise have distinct requirements, and if node.js can’t support those, will never be an option.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[emaste]

@anandsuresh as far as this issue is concerned, if you're finding FreeBSD 12 works fine and 11 or earlier has issues, we should just focus our efforts there.