Git submodule policy / address sanitizer

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Git submodule policy / address sanitizer

Inkscape - Dev mailing list
Dear Inkscape Team,

since gitlib is up and running, the question arizes if we want to use
submodules, and if so which folder structure should we use.

I want to add address / memory sanitizer support and use
https://github.com/arsenm/sanitizers-cmake. It is just a few files and I
could copy them verbatim, but it would make sense to add it as a submodule.

In case you don't know what address/memory sanitizer is: it creates
instrumented builds (about 2x slower) which check for all sorts of
invalid memory usage, like access to freed memory or reading
uninitialized variables. The cmake scripts above enable this by a cmake
define, similar to switching between debug and release builds. In my
experience it saves a lot of time in tracking down obscure bugs in
complex C++ code. I right now have a debug/release deviation I want to
track down.

Best regards,

Michael


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

Jabier Arraiza
Thanks Michael, deviation between debug/release is boring!
Best!

On Sat, 2017-06-10 at 15:28 +0200, Michael Soegtrop via Inkscape-devel
wrote:

> Dear Inkscape Team,
>
> since gitlib is up and running, the question arizes if we want to use
> submodules, and if so which folder structure should we use.
>
> I want to add address / memory sanitizer support and use
> https://github.com/arsenm/sanitizers-cmake. It is just a few files
> and I
> could copy them verbatim, but it would make sense to add it as a
> submodule.
>
> In case you don't know what address/memory sanitizer is: it creates
> instrumented builds (about 2x slower) which check for all sorts of
> invalid memory usage, like access to freed memory or reading
> uninitialized variables. The cmake scripts above enable this by a
> cmake
> define, similar to switching between debug and release builds. In my
> experience it saves a lot of time in tracking down obscure bugs in
> complex C++ code. I right now have a debug/release deviation I want
> to
> track down.
>
> Best regards,
>
> Michael
>
>
> -------------------------------------------------------------------
> -----------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Inkscape-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/inkscape-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

Christoffer Holmstedt
My experience is that git submodules are really cumbersome to work with. Duplication of code is not nice but if sanitizers-cmake is no available as a system package maybe it is just easier to copy the files to Inkscape repository.

2017-06-10 16:45 GMT+02:00 Jabier Arraiza <[hidden email]>:
Thanks Michael, deviation between debug/release is boring!
Best!

On Sat, 2017-06-10 at 15:28 +0200, Michael Soegtrop via Inkscape-devel
wrote:
> Dear Inkscape Team,
>
> since gitlib is up and running, the question arizes if we want to use
> submodules, and if so which folder structure should we use.
>
> I want to add address / memory sanitizer support and use
> https://github.com/arsenm/sanitizers-cmake. It is just a few files
> and I
> could copy them verbatim, but it would make sense to add it as a
> submodule.
>
> In case you don't know what address/memory sanitizer is: it creates
> instrumented builds (about 2x slower) which check for all sorts of
> invalid memory usage, like access to freed memory or reading
> uninitialized variables. The cmake scripts above enable this by a
> cmake
> define, similar to switching between debug and release builds. In my
> experience it saves a lot of time in tracking down obscure bugs in
> complex C++ code. I right now have a debug/release deviation I want
> to
> track down.
>
> Best regards,
>
> Michael
>
>
> -------------------------------------------------------------------
> -----------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Inkscape-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/inkscape-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel




--
Christoffer Holmstedt

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

objarni
Christoffer;

What is bad with git submodules in your experience?


Mvh


/Olof
-----------------
Är du systemutvecklare?
Spana in https://cilamp.se



On 10 June 2017 at 20:12, Christoffer Holmstedt
<[hidden email]> wrote:

> My experience is that git submodules are really cumbersome to work with.
> Duplication of code is not nice but if sanitizers-cmake is no available as a
> system package maybe it is just easier to copy the files to Inkscape
> repository.
>
> 2017-06-10 16:45 GMT+02:00 Jabier Arraiza <[hidden email]>:
>>
>> Thanks Michael, deviation between debug/release is boring!
>> Best!
>>
>> On Sat, 2017-06-10 at 15:28 +0200, Michael Soegtrop via Inkscape-devel
>> wrote:
>> > Dear Inkscape Team,
>> >
>> > since gitlib is up and running, the question arizes if we want to use
>> > submodules, and if so which folder structure should we use.
>> >
>> > I want to add address / memory sanitizer support and use
>> > https://github.com/arsenm/sanitizers-cmake. It is just a few files
>> > and I
>> > could copy them verbatim, but it would make sense to add it as a
>> > submodule.
>> >
>> > In case you don't know what address/memory sanitizer is: it creates
>> > instrumented builds (about 2x slower) which check for all sorts of
>> > invalid memory usage, like access to freed memory or reading
>> > uninitialized variables. The cmake scripts above enable this by a
>> > cmake
>> > define, similar to switching between debug and release builds. In my
>> > experience it saves a lot of time in tracking down obscure bugs in
>> > complex C++ code. I right now have a debug/release deviation I want
>> > to
>> > track down.
>> >
>> > Best regards,
>> >
>> > Michael
>> >
>> >
>> > -------------------------------------------------------------------
>> > -----------
>> > Check out the vibrant tech community on one of the world's most
>> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> > _______________________________________________
>> > Inkscape-devel mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/inkscape-devel
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Inkscape-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/inkscape-devel
>>
>
>
>
> --
> Christoffer Holmstedt
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Inkscape-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/inkscape-devel
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

Yale Zhang
Just a reminder, address sanitizer can miss overflows if a read/write
ends up pointing to another valid memory location. A more fool proof
tool is pointer bounds checking. It's in GCC but requires a Skylake
processor or later.

On Sat, Jun 10, 2017 at 2:01 PM, Olof Bjarnason
<[hidden email]> wrote:

> Christoffer;
>
> What is bad with git submodules in your experience?
>
>
> Mvh
>
>
> /Olof
> -----------------
> Är du systemutvecklare?
> Spana in https://cilamp.se
>
>
>
> On 10 June 2017 at 20:12, Christoffer Holmstedt
> <[hidden email]> wrote:
>> My experience is that git submodules are really cumbersome to work with.
>> Duplication of code is not nice but if sanitizers-cmake is no available as a
>> system package maybe it is just easier to copy the files to Inkscape
>> repository.
>>
>> 2017-06-10 16:45 GMT+02:00 Jabier Arraiza <[hidden email]>:
>>>
>>> Thanks Michael, deviation between debug/release is boring!
>>> Best!
>>>
>>> On Sat, 2017-06-10 at 15:28 +0200, Michael Soegtrop via Inkscape-devel
>>> wrote:
>>> > Dear Inkscape Team,
>>> >
>>> > since gitlib is up and running, the question arizes if we want to use
>>> > submodules, and if so which folder structure should we use.
>>> >
>>> > I want to add address / memory sanitizer support and use
>>> > https://github.com/arsenm/sanitizers-cmake. It is just a few files
>>> > and I
>>> > could copy them verbatim, but it would make sense to add it as a
>>> > submodule.
>>> >
>>> > In case you don't know what address/memory sanitizer is: it creates
>>> > instrumented builds (about 2x slower) which check for all sorts of
>>> > invalid memory usage, like access to freed memory or reading
>>> > uninitialized variables. The cmake scripts above enable this by a
>>> > cmake
>>> > define, similar to switching between debug and release builds. In my
>>> > experience it saves a lot of time in tracking down obscure bugs in
>>> > complex C++ code. I right now have a debug/release deviation I want
>>> > to
>>> > track down.
>>> >
>>> > Best regards,
>>> >
>>> > Michael
>>> >
>>> >
>>> > -------------------------------------------------------------------
>>> > -----------
>>> > Check out the vibrant tech community on one of the world's most
>>> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> > _______________________________________________
>>> > Inkscape-devel mailing list
>>> > [hidden email]
>>> > https://lists.sourceforge.net/lists/listinfo/inkscape-devel
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Inkscape-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/inkscape-devel
>>>
>>
>>
>>
>> --
>> Christoffer Holmstedt
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Inkscape-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/inkscape-devel
>>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Inkscape-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/inkscape-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

Bryce Harrington-3
In reply to this post by Inkscape - Dev mailing list
On Sat, Jun 10, 2017 at 03:28:02PM +0200, Michael Soegtrop via Inkscape-devel wrote:
> Dear Inkscape Team,
>
> since gitlib is up and running, the question arizes if we want to use
> submodules, and if so which folder structure should we use.
>
> I want to add address / memory sanitizer support and use
> https://github.com/arsenm/sanitizers-cmake. It is just a few files and I
> could copy them verbatim, but it would make sense to add it as a submodule.

IMHO, in the interest of keeping things simple while people are climbing
learning curves for git, let's hold off on remotes and submodules for now.

Also, in general, I know we've been apt to pull stuff into the main
codebase but going forward as much as possible let's try to keep
external dependencies completely external, with detection logic in
cmake.  The more we can slim down the codebase the easier it'll be for
new folks to learn and less work it'll be for busy old timers to
maintain over the long run.

Regarding testing tools, improving code quality for Inkscape is
extremely valuable work, so the more the merrier.  We just need to be
cognizant of setting them up as default off, with the appropriate cmake
stuff to detect the test dependency and give the user a config option to
switch them on.

> In case you don't know what address/memory sanitizer is: it creates
> instrumented builds (about 2x slower) which check for all sorts of
> invalid memory usage, like access to freed memory or reading
> uninitialized variables. The cmake scripts above enable this by a cmake
> define, similar to switching between debug and release builds. In my
> experience it saves a lot of time in tracking down obscure bugs in
> complex C++ code. I right now have a debug/release deviation I want to
> track down.

We had set up Coverity at one point, which also checked similar stuff.
I think that's probably defunct currently, though.  We didn't really
have enough people following up on problems for it to work.

The tool almost doesn't matter - the real value is the people who work
on following up on issues the testing finds.  The first run can
typically identify a LOT of work items that'll take a lot of labor to
work through; ongoing runs can be less work but still need folks to
follow up on stuff, and unfortunately a lot of regular developers will
be too busy with other stuff to bother so it helps to have dedicated
people familiar with the testing tool.

So, as a rule of thumb, you'd want to see at least, say, 3 people
committing to watch and follow up on issues found by a particular tool,
if adoption of the tool is going to be invasive in terms of
infrastructure requirements or adjustments to development processes.

Bryce

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

Inkscape - Dev mailing list
Dear Bryce,

> Also, in general, I know we've been apt to pull stuff into the main
> codebase but going forward as much as possible let's try to keep
> external dependencies completely external, with detection logic in
> cmake.  The more we can slim down the codebase the easier it'll be for
> new folks to learn and less work it'll be for busy old timers to
> maintain over the long run.
>
The piece I suggested is exactly that: cmake code which detects and
implements sanitizer support. But it is just 10 or so cmake files and
can also be copied.

>
> So, as a rule of thumb, you'd want to see at least, say, 3 people
> committing to watch and follow up on issues found by a particular tool,
> if adoption of the tool is going to be invasive in terms of
> infrastructure requirements or adjustments to development processes.
>
>

I think the sanitizers are different. It is more a tool to help
developers debug their code in case they see strange behavior like
Debug/Release mode deviations.

Best regards,

Michael


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

Bryce Harrington-3
On Sun, Jun 11, 2017 at 11:51:31PM +0200, Michael Soegtrop wrote:

> Dear Bryce,
>
> > Also, in general, I know we've been apt to pull stuff into the main
> > codebase but going forward as much as possible let's try to keep
> > external dependencies completely external, with detection logic in
> > cmake.  The more we can slim down the codebase the easier it'll be for
> > new folks to learn and less work it'll be for busy old timers to
> > maintain over the long run.
> >
> The piece I suggested is exactly that: cmake code which detects and
> implements sanitizer support. But it is just 10 or so cmake files and
> can also be copied.

If it is entirely inside the cmake directories, then that does remove
most of the worry, although 10 files sounds like a lot of cmake...

> > So, as a rule of thumb, you'd want to see at least, say, 3 people
> > committing to watch and follow up on issues found by a particular tool,
> > if adoption of the tool is going to be invasive in terms of
> > infrastructure requirements or adjustments to development processes.
>
> I think the sanitizers are different. It is more a tool to help
> developers debug their code in case they see strange behavior like
> Debug/Release mode deviations.

They're not different, same concept applies - make sure other people
besides yourself will be committing to use it before you add it to the
repo.

I don't mean to pick on this tool in particular, there's just been a bit
of a history of tests, build systems, frameworks, and so on being
championed by one person but never really catching on beyond them, so
that when their attentions moved on the bits were left in place to
atrophy.

Bryce



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Git submodule policy / address sanitizer

Inkscape - Dev mailing list
In reply to this post by Inkscape - Dev mailing list
On 06/11/2017 04:51 PM, Michael Soegtrop via Inkscape-devel wrote:

> Dear Bryce,
>
>> Also, in general, I know we've been apt to pull stuff into the main
>> codebase but going forward as much as possible let's try to keep
>> external dependencies completely external, with detection logic in
>> cmake.  The more we can slim down the codebase the easier it'll be for
>> new folks to learn and less work it'll be for busy old timers to
>> maintain over the long run.
> The piece I suggested is exactly that: cmake code which detects and
> implements sanitizer support. But it is just 10 or so cmake files and
> can also be copied.
Another solution here would be to have the CI job that did the sanitizer
check download the CMake files on the fly from another source. This way
we wouldn't have to have them in our repo at all, just when the job is
run. We should save them as an artifact of that job for debugging though.

Ted


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel

signature.asc (837 bytes) Download Attachment