Update Clang toolchain from 18.0.0 to 18.1.8 #12365

catenacyber · 2024-08-16T09:48:20Z

Follow-up on #12077 by @alexcrichton cc @maflcko

Main difference is to update infra/base-images/base-runner/profraw_update.py so that oss-fuzz converts profraw version 8 to 9 (and llvm-cov seems more tolerant in older version reading cf llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

This way, it should be more transparent for projects, that can be updated individually or not

catenacyber · 2024-08-16T09:49:31Z

This is rather a draft for discussion and testing : what is the good way to test that ?

maflcko · 2024-08-16T09:54:21Z

Not sure about authorship, but personally, I think it would be easier to follow the history, if you just took the commit 6defef2 from the other pull request. You can then rebase it on master, or just create a merge commit with master (my preference, because it leaves the commit ID exactly intact).

(Edit: Or if you want to keep the pins for now, it'd probably be commit ID 8b26362)

This will also include other useful changes, like unpinning the projects that no longer need the pin.

For testing, you can test locally (probably only works for a few spot tests), or wait for an admin to create the global infra build.

catenacyber · 2024-08-16T10:08:23Z

Not sure about authorship, but personally, I think it would be easier to follow the history, if you just took the commit 6defef2 from the other pull request.

Yes, cherry-picking the commits from Alex :-)

This will also include other useful changes, like unpinning the projects that no longer need the pin.

I am not sure I want to do that now : I want the least changes possible, only ones in infra, not ones in individual projects :

This way, if this patch is good, there should be no changes in the builds status...
Except new warnings introduced in clang 18.1 that can make some compilations fail. In this case, maybe the projects maintainers want to fix these warnings instead of ignoring them (I do as a project maintainer)
And except other surprises

maflcko · 2024-08-22T12:12:14Z

Would be nice to get an infra build here, to see if there are any build issues or other issues

catenacyber · 2024-08-25T20:09:07Z

Would be nice to get an infra build here, to see if there are any build issues or other issues

Should I do something ?

maflcko · 2024-08-26T06:29:16Z

I think only repo admins can trigger the infra trial-build

catenacyber · 2024-08-26T15:36:57Z

cc @jonathanmetzman

DonggeLiu · 2024-08-27T02:09:39Z

/gcbrun trial_build.py all

catenacyber · 2024-08-27T14:42:47Z

@DonggeLiu what is the outcome of the trial build ?

maflcko · 2024-08-27T14:58:55Z

For some reason the logs are truncated now. (Previously they weren't and it was possible to see the tail and the result)

The ones that I see are all failures. (I checked a few URLs)

logs

catenacyber · 2024-08-27T15:02:44Z

Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #1: CommandException: No URLs matched: gs://oss-fuzz-coverage/tinygltf/textcov_reports/20240827/*
Step #1: CommandException: 1 file/object could not be transferred.
Step #6: 334586aa2186: Pull complete
Step #6: 174a75ccacd5: Pull complete
Step #6: Digest: sha256:c7405fed95553b9fb9be1723f233b2c1ec42273135169d98a9c1a4da4344fafa
Step #6: Status: Downloaded newer image for gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Step #6: gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Finished Step #6
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/gsutil" failed: step exited with non-zero status: 1

What is going on here ?

DonggeLiu · 2024-08-28T00:55:14Z

For some reason the logs are truncated now. (Previously they weren't and it was possible to see the tail and the result)
The ones that I see are all failures. (I checked a few URLs)

Thanks @maflcko. This is the full log.

I also noticed many failures.

DonggeLiu · 2024-08-28T01:09:32Z

Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #1: CommandException: No URLs matched: gs://oss-fuzz-coverage/tinygltf/textcov_reports/20240827/*
Step #1: CommandException: 1 file/object could not be transferred.
Step #6: 334586aa2186: Pull complete
Step #6: 174a75ccacd5: Pull complete
Step #6: Digest: sha256:c7405fed95553b9fb9be1723f233b2c1ec42273135169d98a9c1a4da4344fafa
Step #6: Status: Downloaded newer image for gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Step #6: gcr.io/oss-fuzz-base/base-builder-swift-testing-profraw9-update:latest
Finished Step #6
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/gsutil" failed: step exited with non-zero status: 1

What is going on here ?

@oliverchang, not sure if this is a known issue:
The missing report error implies that trail_build should have fetched yesterday's report, not today's.
I triggered the experiment on the morning of August 27 (Sydney), and it searches for a report in the directory 0827/. However, the report has not been generated yet.
I am pretty sure the coverage report exists now at:
https://storage.googleapis.com/oss-fuzz-coverage/tinygltf/textcov_reports/20240827/fuzz_gltf.covreport

Before we fix the bug, we should run the command after 6 PM (Syndey), given that the report was generated after 5 PM:

catenacyber · 2024-09-01T19:32:35Z

@DonggeLiu will you run the command again with the good timing ?

DonggeLiu · 2024-09-01T23:45:01Z

@DonggeLiu will you run the command again with the good timing ?

Sure, I will do it in the afternoon (hopefully I can still remember this).

DonggeLiu · 2024-09-03T12:25:14Z

/gcbrun trial_build.py all

maflcko · 2024-09-04T06:39:45Z

For reference, the output is:

trial-build

Remaining builds: 20/1007, defaultdict(<class 'list'>, {'apache-poi': ['8111fa89-ffae-406f-a0a3-9e030531718c'], 'aspell': ['2212a2a7-e36e-4d0e-bd43-8a431d6e5458'], 'binutils': ['33a8c9f8-a2c5-4c7c-b56b-4b6b7cacd2f7'], 'cloud-custodian': ['6ee0ab2c-c6da-4147-b0f0-893aba501f45'], 'cmake': ['bcbab0f6-4b6a-41a0-9595-097b5ea9b643'], 'duckdb': ['3d2f04d6-d453-4815-af78-2078c51d1e5d'], 'gitpython': ['ca5d9ed0-2494-44df-8985-26c774af3203'], 'ipaddress': ['fec49ceb-49da-4e62-a736-05c6803e7dfb'], 'jansi': ['fafb268c-e9d2-468f-86f2-2a4dd7112580'], 'kubeedge': ['878a7656-4f2f-43d5-93da-7ec019bb58d8'], 'onednn': ['5c18d180-53f2-4a89-b1e5-ce220ef40404'], 'opencsv': ['f5ebe13a-5f5a-4911-8a5d-cd9ac0e53a04'], 'openweave': ['9589d151-fac6-415e-b56f-d10534d276d3'], 'pasta': ['2d33ffc1-f266-4821-9724-fe5ceb39913a'], 'simd': ['a4389c9d-ae8a-4771-a089-d81e308b78e0', '2e0e616b-049b-41f3-9609-08f2abfe3680'], 'sqlalchemy_jsonfield': ['29357d3d-f01f-4742-ae70-6cabf3ae1704'], 'utf8proc': ['9620eec1-fbd4-41ed-870b-7e348ec8b36b'], 'xen': ['efe68fc4-1aad-497b-9980-8b5ca2bfbfb8'], 'zlib-ng': ['82469efd-1f99-4bb7-a73b-bccdfc1102b1'], 'llvm': ['e2cb5d7e-8647-482c-9936-82e859b00004']}).

maflcko · 2024-09-04T06:41:13Z

It looks like most of the failures are due to the branch being outdated.

Before a trial-build, my recommendation would be to merge the current master branch into the pull request. Otherwise, the result may be inaccurate and outdated.

DonggeLiu · 2024-09-04T12:14:06Z

/gcbrun trial_build.py all

catenacyber · 2024-09-04T12:31:41Z

Failed builds: 38/1007

Looked already pretty good, did it not ?

DonggeLiu · 2024-09-04T23:52:10Z

Here is the trial build log log.

I (or someone else) will take a look into them

maflcko · 2024-09-05T04:53:29Z

I looked at the result and I think some projects break due to the compiler bump. It would be good to pin them, or otherwise fix them.

https://github.com/google/oss-fuzz/pull/12365/checks?check_run_id=29667299742

trial-build

Remaining builds: 31/1011, defaultdict(<class 'list'>, {'bitcoin-core': ['e97dac9b-220c-45ee-8343-84e9eec6bfe1'], 'bluez': ['b1499b2f-148a-4a43-af5e-084863c2c8a9'], 'checkstyle': ['8d87523f-7b24-473e-af3a-3bdcd241c6ee'], 'cloud-custodian': ['e38a48e3-419c-4e2b-9dd7-f94f18419a20'], 'cpuinfo': ['54fa5e3d-885d-4a6e-8a77-32fd3b80213b'], 'duckdb': ['b0426bc7-3e54-403b-a847-56ee0cb9a412'], 'ffmpeg': ['ac82d2f0-d7ca-4e70-b08b-b660c9dc2a04'], 'genshi': ['d2f6169b-2ec1-47df-8557-505f95afaa7b'], 'gprof2dot': ['33736e00-d147-4d0d-b2d8-7b9f7b78e833'], 'hikaricp': ['0958c36c-12ab-4ecb-9816-19b5e45cfd10'], 'htslib': ['61e7a9d0-2b54-4d1e-a96e-e072703a63f8'], 'http-parser': ['7dc60deb-9c57-4d76-825d-5ed27c6e6d74'], 'imagemagick': ['4075bbc0-1eb4-443a-b141-828a5cea34fe'], 'jq': ['f997d55a-3b45-46ab-b6aa-7f1b8a0888ab'], 'libreoffice': ['021c4098-7db3-49a5-ade8-535674b6fa3d'], 'libxslt': ['2f7b1a83-51da-4096-a39a-5216ee44b14e'], 'libyal': ['c4323b37-191e-4289-bc75-3b3c41a7ec79'], 'muparser': ['7fd96760-4d21-44a5-ae01-cf24e3af2a60'], 'neomutt': ['ab316d4f-3f3b-44cf-a37e-2c7ce2e0a3fb'], 'onednn': ['e6448e28-040b-445b-a7aa-9e8fdb06e382'], 'openweave': ['72af842b-1ce3-45f8-b8d5-3b1202183974'], 'python-phonenumbers': ['229da2ce-22f8-4c23-a3f3-e5fda4a17b1e'], 'simd': ['0d3301bd-6bc7-461c-8722-9fa3ae920b8a', '50e5284d-c067-4800-b6fc-a66b0641243a'], 'spring-cloud-config': ['d6f33c33-d6dd-4e0c-8011-1047e1f9be71'], 'sqlalchemy_jsonfield': ['c2b8ae95-517c-4152-ae49-9bae934a762e'], 'toolbelt': ['f71f41d6-59ec-4744-9dc7-e4656c7819ef'], 'vorbis': ['7a26d044-8d1d-4b68-9327-edb71f387ef3'], 'xmlpull': ['c72c9058-a0f0-408f-98cc-5a68d2b5150d'], 'yoga': ['2ef90f49-a3f6-4036-b0d5-6f00e465cfa3'], 'zopfli': ['10bee761-eaa3-4727-974b-9315b59786f0'], 'llvm': ['5346b641-4627-48a2-aa21-cdf3b33cca3b']}).

catenacyber · 2024-09-05T14:32:54Z

First analysis :

digest : network error
envoy: new warning Wthread-safety-reference-return
freeimage: network error
icu: centipede build failure, is it related ?
librawspeed: new warning Wnan-infinity-disabled
libucl: strange, maybe due to timeouts and stack overflows in corpus for coverage
libxml2: looks like incompatibility between trial build and aarch64

I looked at the result and I think some projects break due to the compiler bump. It would be good to pin them, or otherwise fix them.

I guess the new warnings need to be fixed by each project

Only 2 out of the 7 first failures (icu and libucl) I do not understand and they may be due to this PR.
I think this PR will fix more than it breaks, would it not ?

maflcko · 2024-09-05T15:52:09Z

Only 2 out of the 7 first failures (icu and libucl) I do not understand and they may be due to this PR.

Sometimes the project starts to fail to build at the same time the infra run is scheduled. So it may or may not be related to the PR and another build has to be done in those cases (either locally, or in another infra run).

If the project is failing, I think maintainers prefer it to be pinned or fixed. Pinning can be achieved similar to commit 767c7d3. (You'll have to adjust the sha256 to be a recent one).

catenacyber · 2024-09-05T19:09:36Z

mpv : centipede again...
poppler: unrelated Could NOT find Boost
rust-lexical: build failure to investigate
samba: Wformat-truncation intended by some test
simdutf: aarch64
suricata: I guess I need to update it for rust and C compiler to match
tinyusb: aarch64

catenacyber · 2024-09-05T19:19:09Z

tomlkit: network error ?
wasmer: new rust warnings
zeek: looks like I need to fix coverage for targets generating multiple profraw in one run ?..
Hadoop: looks unrelated
lua: aarch64
tarantool: aarch64

try to make it more clear which changes are needed for which version

catenacyber · 2024-09-25T12:01:35Z

By the way, I also see that log4j2, pcre2 had recent changes in master

catenacyber · 2024-09-26T13:52:10Z

@jonathanmetzman what do you think about this PR ?

jonathanmetzman · 2024-09-27T16:04:08Z

merge on monday if this passes except for arm

jonathanmetzman · 2024-09-27T16:04:31Z

/gcbrun trial_build.py apache-commons-lang arrow-java cert-manager cyclonedds elfutils libxml2 monero pcre2 s2geometry simdutf tinyusb util-linux log4j2 rust-coreutils lua tarantool

catenacyber · 2024-09-29T18:38:58Z

merge on monday if this passes except for arm

Looks good :-)
Step #1: Failed builds: 5/14, {'tinyusb': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-20a40feb-1114-43da-bb13-3b774d36de10.txt', 'simdutf': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-a1da1a0d-95ec-49a7-82e2-be3e10980bd9.txt', 'libxml2': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-9c64211f-6ec3-4106-a4a4-c18ee20ea182.txt', 'tarantool': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-6d988b57-f825-4583-a218-6a5ba9c69d93.txt', 'lua': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5335f04e-2db6-474d-bddb-e48ddf4efbc1.txt'}
Finished Step #1

catenacyber · 2024-10-01T08:01:37Z

merge on monday if this passes except for arm

@jonathanmetzman will you merge today ?

evverx · 2024-10-01T20:21:30Z

projects/elfutils/Dockerfile

-FROM gcr.io/oss-fuzz-base/base-builder
+FROM gcr.io/oss-fuzz-base/base-builder@sha256:56905c98ae0083d14da0e7371184e694560a74750533f321ac0e9145af0e8d2e
+# ! This project was pinned after a clang bump. Please remove the pin, Try to fix any build warnings and errors, as well as runtime errors
+# see https://github.com/google/oss-fuzz/pull/12365


I'm not sure it's the best way to notify projects. Though given that the latest build failure wasn't reported and the primary contact pointing to the elfutils mailing doesn't seem to work anymore bug reports are likely to go unnoticed as well.

Either way elftuils is built with clang version 18.1.8 (Fedora 18.1.8-1.fc40) shipped by Fedora just fine. Could it be that something else is broken in the OSS-Fuzz images? The backtrace I saw points to #include <iostream> and that's weird.

I opened #12550 to trigger the CI but it doesn't look like the base-builder image has been updated yet.

I'm not sure it's the best way to notify projects

Me neither, but that was asked instead of letting the projects fail because of warnings or such...

that was asked instead of letting the projects fail because of warnings or such

I agree that it's better than the alternatives.

Now that the image is rolled out it looks like the failure is flaky:
https://github.com/google/oss-fuzz/actions/runs/11145617408 (where the coverage build failed)
https://github.com/google/oss-fuzz/actions/runs/11145449506 (where the coverage and UBSan builds failed)

I reproduced it more or less reliably in https://github.com/google/oss-fuzz/actions/runs/11146061465 (where all the relevant builds failed) by removing -j from make. The backtrace seems to point to #include <iostream> and I'm not sure why yet.

I'm 99% sure that it's an OSS-Fuzz toolchain bug. I was curious where stack came from and it turns out it's from src/.deps/srcfiles.Tpo (which is used when srcfiles.cxx is compiled):

$ grep stack src/.deps/srcfiles.Tpo /usr/local/bin/../include/c++/v1/deque stack \ stack:

and it should probably be /usr/local/bin/../include/c++/v1/stack. I'm guessing elfutils is unlucky in that it has the stack binary (which is rejected with all those UTF-8 warnings because it is a binary).

srcfiles.cxx started failing to compile with the OSS-Fuzz toolchain when it was switched from clang-18.0.0 to clang-18.1.8 in google#12365. google#12365 (comment) It's probably an OSS-Fuzz toolchain bug but it doesn't matter much because the srcfiles binary isn't relevant in terms of fuzzing and can safely be excluded.

* [elfutils] unpin the base-builder image * [elfutils] show clang version to make it a bit easier to figure out what's going on. * [elfutils] no longer compile srcfiles.cxx srcfiles.cxx started failing to compile with the OSS-Fuzz toolchain when it was switched from clang-18.0.0 to clang-18.1.8 in #12365. #12365 (comment) It's probably an OSS-Fuzz toolchain bug but it doesn't matter much because the srcfiles binary isn't relevant in terms of fuzzing and can safely be excluded.

A draft following #12365 and plan to cherry-pick e0d6875

Following #12365 : only zeek and opencv seem fixed Co-authored-by: Alex Crichton <[email protected]>

catenacyber · 2024-10-03T07:09:41Z

@FrankYFTang for your information, with recent clang bump, icu had centipede disabled. You can try to reenable it

catenacyber · 2024-10-03T07:11:28Z

@Alexhuszagh with recent clang and rust bump, rust-lexical is pinned to an old builder, will you try to fix it ?

catenacyber · 2024-10-03T07:12:38Z

@manunio with recent clang and rust bump, wasmer uses an old nightly (because of dead code warnings with nightly-2024-07-12), would you try to fix them ?

manunio · 2024-10-03T10:45:56Z

@manunio with recent clang and rust bump, wasmer uses an old nightly (because of dead code warnings with nightly-2024-07-12), would you try to fix them ?

Thanks for the ping, I will look into it.

This unpins the image for `rust-lexical`, due to patches in the fuzz targets Cargo.toml to remove LTO from the release profile. Related to: - Alexhuszagh/rust-lexical/issues/164 - Alexhuszagh/rust-lexical/issues/166 - #12365

@maflcko

Following #12365 cc @maflcko

catenacyber mentioned this pull request Aug 16, 2024

Support for LLVM 18 fuzzing #11626

Closed

catenacyber force-pushed the profraw9-update branch from 171b315 to dd7bf2b Compare August 16, 2024 10:01

maflcko mentioned this pull request Aug 20, 2024

[infra] Unpin clang version #12385

Closed

maflcko mentioned this pull request Aug 27, 2024

Rust build image needs updating #12410

Open

catenacyber added 3 commits September 25, 2024 14:00

envoy: pin base-builder before clang18 bump

2b59f49

infra/coverage: siplification for profraw_update

695666b

try to make it more clear which changes are needed for which version

elfutils: pin base-builder before clang18 bump

4066c8c

catenacyber force-pushed the profraw9-update branch from 177e875 to 4066c8c Compare September 25, 2024 12:00

vitorguidi merged commit 1778ebe into google:master Oct 1, 2024
12 of 19 checks passed

evverx reviewed Oct 1, 2024

View reviewed changes

This was referenced Oct 2, 2024

Clang 18.1 remove rust pins #12555

Merged

Remove some @sha256:... pins in Dockerfiles #12560

Merged

remove usage of OLD_LLVMPASS #12561

Merged

evverx mentioned this pull request Oct 2, 2024

[elfutils] unpin the base-builder image #12550

Merged

DavidKorczynski pushed a commit that referenced this pull request Oct 2, 2024

Clang 18.1 remove rust pins (#12555)

1e451bf

A draft following #12365 and plan to cherry-pick e0d6875

DavidKorczynski pushed a commit that referenced this pull request Oct 2, 2024

Remove some @sha256:... pins in Dockerfiles (#12560)

341b2fd

Following #12365 : only zeek and opencv seem fixed Co-authored-by: Alex Crichton <[email protected]>

catenacyber mentioned this pull request Oct 3, 2024

rust: unpin serde OISF/suricata#11871

Closed

Alexhuszagh mentioned this pull request Oct 3, 2024

[CI] Update Rust-Lexical Version on OSS-Fuzz Alexhuszagh/rust-lexical#164

Closed

catenacyber mentioned this pull request Oct 3, 2024

rust: unpin serde OISF/suricata#11872

Closed

Alexhuszagh mentioned this pull request Oct 4, 2024

Unpin rust-lexical image version. #12564

Merged

DavidKorczynski pushed a commit that referenced this pull request Oct 9, 2024

remove usage of OLD_LLVMPASS (#12561)

aebe4a9

Following #12365 cc @maflcko

Update Clang toolchain from 18.0.0 to 18.1.8 #12365

Update Clang toolchain from 18.0.0 to 18.1.8 #12365

Conversation

catenacyber commented Aug 16, 2024

catenacyber commented Aug 16, 2024

maflcko commented Aug 16, 2024 • edited Loading

catenacyber commented Aug 16, 2024

maflcko commented Aug 22, 2024

catenacyber commented Aug 25, 2024

maflcko commented Aug 26, 2024

catenacyber commented Aug 26, 2024

DonggeLiu commented Aug 27, 2024

catenacyber commented Aug 27, 2024

maflcko commented Aug 27, 2024 • edited Loading

catenacyber commented Aug 27, 2024

DonggeLiu commented Aug 28, 2024

DonggeLiu commented Aug 28, 2024

catenacyber commented Sep 1, 2024

DonggeLiu commented Sep 1, 2024

DonggeLiu commented Sep 3, 2024

maflcko commented Sep 4, 2024

maflcko commented Sep 4, 2024

DonggeLiu commented Sep 4, 2024

catenacyber commented Sep 4, 2024

DonggeLiu commented Sep 4, 2024

maflcko commented Sep 5, 2024

catenacyber commented Sep 5, 2024

maflcko commented Sep 5, 2024

catenacyber commented Sep 5, 2024 • edited Loading

catenacyber commented Sep 5, 2024

catenacyber commented Sep 25, 2024

catenacyber commented Sep 26, 2024

jonathanmetzman commented Sep 27, 2024

jonathanmetzman commented Sep 27, 2024

catenacyber commented Sep 29, 2024

catenacyber commented Oct 1, 2024

evverx Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

catenacyber Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

evverx Oct 2, 2024

Choose a reason for hiding this comment

evverx Oct 2, 2024

Choose a reason for hiding this comment

catenacyber commented Oct 3, 2024

catenacyber commented Oct 3, 2024

catenacyber commented Oct 3, 2024

manunio commented Oct 3, 2024

maflcko commented Aug 16, 2024 •

edited

Loading

maflcko commented Aug 27, 2024 •

edited

Loading

catenacyber commented Sep 5, 2024 •

edited

Loading

evverx Oct 1, 2024 •

edited

Loading

catenacyber Oct 2, 2024 •

edited

Loading