Wacom smartpad support - request for ideas

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Wacom smartpad support - request for ideas

Peter Hutterer
Benjamin Tissoires and I have been working on a project called Tuhi that
communicates with the wacom smartpad devices and pulls the data off them.

https://github.com/tuhiproject/tuhi/

Devices working so far are the Bamboo Spark, Slate and the Intuos Pro Paper.
Smartpad devices aren't traditional graphics tablets [1] but instead have a
paper notepad on top of a sensor with a special pen. You draw on the paper,
the sensor registers the drawings and stores it in a custom vector-based
format in the device. Later, you connect the device over bluetooth to the
host and fetch the drawings from the firmware.

This is the bit that tuhi does, and it exposes the drawings as a simple
json-based vector format overa session DBus API. To interact, you basically
connect to the dbus daemon, tell it to listen to the device and then get
notified about new drawings. Then you can fetch the drawings one-by-one as
needed.

Tuhi only has a commandline client for debugging and we don't intend to ship
that client, it's for debugging only. Eventually, we may have a standalone
application similar to Wacom's Inkspace (yeah, confusing, isn't it?). This
app would be a gallery-style application that displays the drawings and
exports them to the desired formats. That's a fairly straightforward
process.

But right now what we're interested in is whether inkscape has interest in a
plugin-style implementation that talks to the tuhi daemon. Or whether there
are use-cases where the standalone application is insufficient and some
tighter integration is needed. Or other use-case you can think of that we
haven't yet.

Any comments appreciated!

Cheers,
   Peter


[1] the intuos pro paper is when you plug it in, let's ignore that for now

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Wacom smartpad support - request for ideas

Martin Owens-2
Hi Peter,

This is an exciting addition.

The way I would think about this problem is "what operating systems do
you want to target", if the target is predominately Linux, then the
most super-amazing intergration would be a fuse-fs user space mount.

You plug the device in and it pops up just like a regular usb stick,
but with it's own wacom icon and relivant name. Inside you can offer up
raw or converted file formats, it's up to you how complex you want to
go. Although offering svg files directly would be double rainbow
amazing tbh.

That way Inkscape would be able to open up the files as is, no special
arrangements. The best part is that the files would work with many-many
programs and you get a many-to-many win-win. I highly recommend this.
(some good references for fuse mounts can be found in the google docs
fs, the android adb fs, the media-fs for phones etc)

It doesn't sound like you're targeting MacOSX or Windows considering
you're using DBus. So I hope this is the way you're going to go with
it.

Best Regards, Martin Owens

On Wed, 2018-02-21 at 11:38 +1000, Peter Hutterer wrote:

> Benjamin Tissoires and I have been working on a project called Tuhi
> that
> communicates with the wacom smartpad devices and pulls the data off
> them.
>
> https://github.com/tuhiproject/tuhi/
>
> Devices working so far are the Bamboo Spark, Slate and the Intuos Pro
> Paper.
> Smartpad devices aren't traditional graphics tablets [1] but instead
> have a
> paper notepad on top of a sensor with a special pen. You draw on the
> paper,
> the sensor registers the drawings and stores it in a custom vector-
> based
> format in the device. Later, you connect the device over bluetooth to
> the 
> host and fetch the drawings from the firmware.
>
> This is the bit that tuhi does, and it exposes the drawings as a
> simple
> json-based vector format overa session DBus API. To interact, you
> basically
> connect to the dbus daemon, tell it to listen to the device and then
> get
> notified about new drawings. Then you can fetch the drawings one-by-
> one as
> needed.
>
> Tuhi only has a commandline client for debugging and we don't intend
> to ship
> that client, it's for debugging only. Eventually, we may have a
> standalone
> application similar to Wacom's Inkspace (yeah, confusing, isn't it?).
> This
> app would be a gallery-style application that displays the drawings
> and
> exports them to the desired formats. That's a fairly straightforward
> process.
>
> But right now what we're interested in is whether inkscape has
> interest in a
> plugin-style implementation that talks to the tuhi daemon. Or whether
> there
> are use-cases where the standalone application is insufficient and
> some
> tighter integration is needed. Or other use-case you can think of
> that we
> haven't yet.
>
> Any comments appreciated!
>
> Cheers,
>    Peter
>
>
> [1] the intuos pro paper is when you plug it in, let's ignore that
> for now
>
> -------------------------------------------------------------------
> -----------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Inkscape-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/inkscape-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Wacom smartpad support - request for ideas

Peter Hutterer
Hi Martin,

thanks for your reply, I realised I was a bit scant on details. First, a
basic technical detail: the device is a bluetooth LE device and needs to be
explicitly synced, it doesn't stay connected. Any data that is synced off
the device is deleted by the firmware, thus local buffering is needed
anyway. This buffering is currently handled by the Tuhi daemon, but it's not
a permanent storage solution (we automatically delete old drawings).

On Tue, Feb 27, 2018 at 08:28:06AM -0500, Martin Owens wrote:
> Hi Peter,
>
> This is an exciting addition.
>
> The way I would think about this problem is "what operating systems do
> you want to target", if the target is predominately Linux, then the
> most super-amazing intergration would be a fuse-fs user space mount.

only Linux, because on other platforms Wacom provides applications to
extract the data anyway. There is no need for a daemon on other platforms.

IMO a fuse-fs would be overkill. The data is already synced and buffered,
simply saving this to a preconfigured folder seems sufficient and gets you
99% of the way there. The fuse FS would only get you the look of a USB
stick, at the cost of wrapping everything into a fake fs.

Also, to get the device to sync, you usually have to press the button on the
device and that would require some prompting.

> You plug the device in and it pops up just like a regular usb stick,
> but with it's own wacom icon and relivant name. Inside you can offer up
> raw or converted file formats, it's up to you how complex you want to
> go. Although offering svg files directly would be double rainbow
> amazing tbh.

Offering SVG files is trivial, we already do so in our debugging
tools (minus things like pressure handling, not sure how do do that). But I
just offering SVG means we're losing some of the capabilities: the data we
get from the device however can also include stroke timestamps and per-point
pressure. The Wacom Android app has a slider where you can replay the
drawing and export a partial one. This would be possible with the raw data
but not with an SVG unless we shove it into magic attributes somewhere
(how??).

> That way Inkscape would be able to open up the files as is, no special
> arrangements. The best part is that the files would work with many-many
> programs and you get a many-to-many win-win. I highly recommend this.
> (some good references for fuse mounts can be found in the google docs
> fs, the android adb fs, the media-fs for phones etc)
 
We're planning to have an application that provides SVG export etc. as
standalone anyway. This can take over the user prompting and (background?)
sync anyway, so the files will be available somewhere, ready for the
taking :)


Cheers,
   Peter

> It doesn't sound like you're targeting MacOSX or Windows considering
> you're using DBus. So I hope this is the way you're going to go with
> it.
>
> Best Regards, Martin Owens
>
> On Wed, 2018-02-21 at 11:38 +1000, Peter Hutterer wrote:
> > Benjamin Tissoires and I have been working on a project called Tuhi
> > that
> > communicates with the wacom smartpad devices and pulls the data off
> > them.
> >
> > https://github.com/tuhiproject/tuhi/
> >
> > Devices working so far are the Bamboo Spark, Slate and the Intuos Pro
> > Paper.
> > Smartpad devices aren't traditional graphics tablets [1] but instead
> > have a
> > paper notepad on top of a sensor with a special pen. You draw on the
> > paper,
> > the sensor registers the drawings and stores it in a custom vector-
> > based
> > format in the device. Later, you connect the device over bluetooth to
> > the 
> > host and fetch the drawings from the firmware.
> >
> > This is the bit that tuhi does, and it exposes the drawings as a
> > simple
> > json-based vector format overa session DBus API. To interact, you
> > basically
> > connect to the dbus daemon, tell it to listen to the device and then
> > get
> > notified about new drawings. Then you can fetch the drawings one-by-
> > one as
> > needed.
> >
> > Tuhi only has a commandline client for debugging and we don't intend
> > to ship
> > that client, it's for debugging only. Eventually, we may have a
> > standalone
> > application similar to Wacom's Inkspace (yeah, confusing, isn't it?).
> > This
> > app would be a gallery-style application that displays the drawings
> > and
> > exports them to the desired formats. That's a fairly straightforward
> > process.
> >
> > But right now what we're interested in is whether inkscape has
> > interest in a
> > plugin-style implementation that talks to the tuhi daemon. Or whether
> > there
> > are use-cases where the standalone application is insufficient and
> > some
> > tighter integration is needed. Or other use-case you can think of
> > that we
> > haven't yet.
> >
> > Any comments appreciated!
> >
> > Cheers,
> >    Peter
> >
> >
> > [1] the intuos pro paper is when you plug it in, let's ignore that
> > for now
> >
> > -------------------------------------------------------------------
> > -----------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Inkscape-devel mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/inkscape-devel

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Wacom smartpad support - request for ideas

Martin Owens-2
Hi Peter,

> thanks for your reply, I realised I was a bit scant on details.
> First, a
> basic technical detail: the device is a bluetooth LE device and needs
> to be
> explicitly synced, it doesn't stay connected. Any data that is synced
> off
> the device is deleted by the firmware, thus local buffering is needed
> anyway. This buffering is currently handled by the Tuhi daemon, but
> it's not
> a permanent storage solution (we automatically delete old drawings).

Yes I read the bluetooth part of it after the fact. tbh the whole
exercise would probably need to be ux tested anyway, since a lot of
these types of things can be quite cumbersom if not.

> IMO a fuse-fs would be overkill. The data is already synced and
> buffered,
> simply saving this to a preconfigured folder seems sufficient and
> gets you
> 99% of the way there. The fuse FS would only get you the look of a
> USB
> stick, at the cost of wrapping everything into a fake fs.

Not much of a cost considering the UX gains you get to have. i.e. not
having a seperate program you have to run in order to do things.

> Also, to get the device to sync, you usually have to press the button
> on the device and that would require some prompting.

I see the button as the jumping off point. You press the button, the
demon code is already running with bluez or similar and starting
syncing up and boots up the fusefs as the results page. Would be cool
if I could just open up my laptop, log in and press the physical device
button to get access to my files.

> Offering SVG files is trivial, we already do so in our debugging
> tools (minus things like pressure handling, not sure how do do that).
> But I
> just offering SVG means we're losing some of the capabilities: the
> data we
> get from the device however can also include stroke timestamps and
> per-point
> pressure. The Wacom Android app has a slider where you can replay the
> drawing and export a partial one. This would be possible with the raw
> data
> but not with an SVG unless we shove it into magic attributes
> somewhere
> (how??).

Yeah, stick the timestamp into stroke-time attribute on the path, not
ideal since it's out of spec for svg, but with svg on it's knees, it's
time to add some custom specs anyway. Might be a fun addition to
inkscape later on to add the whole draw forwards/backwards thing and an
interesting saved-undo.

The part that concerns me is the pressure information, since we'd just
have strokes and have no real format for saving powerstrokes as svg
directly. We could convert things to powerstrokes, but there'd have to
be a bit of code that could detect that they're not 'filled out' and do
that work on load.

> We're planning to have an application that provides SVG export etc.
> as
> standalone anyway. This can take over the user prompting and
> (background?)
> sync anyway, so the files will be available somewhere, ready for the
> taking :)

That's the ticket. Just less interaction and more 'just pop the stuff
up' and also less UI work on your side. I suppose you could just make a
folder stick some files in it and pop up a nautilus window, but it's
not as cool as the device recognision you'd get otherwise.

Thanks for taking the time to fill in a few details.

Best Regards, Martin Owens

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Wacom smartpad support - request for ideas

Peter Hutterer
On Tue, Feb 27, 2018 at 06:51:19PM -0500, Martin Owens wrote:

> Hi Peter,
>
> > thanks for your reply, I realised I was a bit scant on details.
> > First, a
> > basic technical detail: the device is a bluetooth LE device and needs
> > to be
> > explicitly synced, it doesn't stay connected. Any data that is synced
> > off
> > the device is deleted by the firmware, thus local buffering is needed
> > anyway. This buffering is currently handled by the Tuhi daemon, but
> > it's not
> > a permanent storage solution (we automatically delete old drawings).
>
> Yes I read the bluetooth part of it after the fact. tbh the whole
> exercise would probably need to be ux tested anyway, since a lot of
> these types of things can be quite cumbersom if not.
>
> > IMO a fuse-fs would be overkill. The data is already synced and
> > buffered,
> > simply saving this to a preconfigured folder seems sufficient and
> > gets you
> > 99% of the way there. The fuse FS would only get you the look of a
> > USB
> > stick, at the cost of wrapping everything into a fake fs.
>
> Not much of a cost considering the UX gains you get to have. i.e. not
> having a seperate program you have to run in order to do things.

following up from that, I started with the fuse code this morning, just to
get a grasp on how things could work. And I almost immediately ran into a
bunch of UX issues that make me question the fuse FS approach again.

First, for a device to sync with a host, it needs to be 'registered' first.
This includes holding the button down for 6+ seconds until the LED starts
blinking, then pressing the button to confirm once the daemon has connected.
The tuhi daemon has the required DBus signals so a client can time this
correctly and show the required notifications. But in an FS we're pretty
much limited to having a README.txt that spells out what you need to do.
Less than ideal...

Second, a device may only sync when the button is pressed. While we can
dbus-activate the tuhi daemon on the first FS access, we cannot communicate
this to the user beyond having a README.txt.

Third, the registration process is a bit whacky and if a user syncs the
device with a different application (like the Android app), the device needs
to be re-registered because its UUID will change. Again, something we have
signals for in the DBus API but not something easy to handle in an FS.

Fourth, this obviously only works when bluetooth is on. If it's off, we need
to ask the user what they want to do.

You can get around these by using the org.freedesktop.Notifications API but
I'm not sure how useful that's going to be.

Then there is the subject of permanent storage: the device deletes drawings
immediately after sync. the tuhi daemon will delete old drawings on restart,
so the only ones guaranteed to be available are the ones drawing in the last
session. This make background sync risky, the files need to be fetched,
converted and saved somewhere in a normal file system for permanence.

Now the fuse fs is little more than a passthrough from the real directory
where everything sits, except that it has a terrible UI :)

so yeah, not convinced the fuse approach is a good one for this particular
device. as it looks like, we'll need a fair bit of UI around it anyway.

Cheers,
   Peter


> > Also, to get the device to sync, you usually have to press the button
> > on the device and that would require some prompting.
>
> I see the button as the jumping off point. You press the button, the
> demon code is already running with bluez or similar and starting
> syncing up and boots up the fusefs as the results page. Would be cool
> if I could just open up my laptop, log in and press the physical device
> button to get access to my files.
>
> > Offering SVG files is trivial, we already do so in our debugging
> > tools (minus things like pressure handling, not sure how do do that).
> > But I
> > just offering SVG means we're losing some of the capabilities: the
> > data we
> > get from the device however can also include stroke timestamps and
> > per-point
> > pressure. The Wacom Android app has a slider where you can replay the
> > drawing and export a partial one. This would be possible with the raw
> > data
> > but not with an SVG unless we shove it into magic attributes
> > somewhere
> > (how??).
>
> Yeah, stick the timestamp into stroke-time attribute on the path, not
> ideal since it's out of spec for svg, but with svg on it's knees, it's
> time to add some custom specs anyway. Might be a fun addition to
> inkscape later on to add the whole draw forwards/backwards thing and an
> interesting saved-undo.
>
> The part that concerns me is the pressure information, since we'd just
> have strokes and have no real format for saving powerstrokes as svg
> directly. We could convert things to powerstrokes, but there'd have to
> be a bit of code that could detect that they're not 'filled out' and do
> that work on load.
>
> > We're planning to have an application that provides SVG export etc.
> > as
> > standalone anyway. This can take over the user prompting and
> > (background?)
> > sync anyway, so the files will be available somewhere, ready for the
> > taking :)
>
> That's the ticket. Just less interaction and more 'just pop the stuff
> up' and also less UI work on your side. I suppose you could just make a
> folder stick some files in it and pop up a nautilus window, but it's
> not as cool as the device recognision you'd get otherwise.
>
> Thanks for taking the time to fill in a few details.
>
> Best Regards, Martin Owens

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Wacom smartpad support - request for ideas

Martin Owens-2
On Thu, 2018-03-01 at 09:29 +1000, Peter Hutterer wrote:
> So yeah, not convinced the fuse approach is a good one for this
> particular device. as it looks like, we'll need a fair bit of UI
> around it anyway.

Thank you for trying the idea, that's more effort than I deserved or
expected. :-)

I'm sorely tempted to buy one of these things to hack on it to be
honest.

If you're producing svgs as the end of your process, then you can have
inkscape open them. Accommodations can be made if we want Inkscape to
be aware of any new attributes you add in to retain useful data.

Best Regards, Martin Owens

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Wacom smartpad support - request for ideas

Peter Hutterer
On Wed, Feb 28, 2018 at 08:03:03PM -0500, Martin Owens wrote:
> On Thu, 2018-03-01 at 09:29 +1000, Peter Hutterer wrote:
> > So yeah, not convinced the fuse approach is a good one for this
> > particular device. as it looks like, we'll need a fair bit of UI
> > around it anyway.
>
> Thank you for trying the idea, that's more effort than I deserved or
> expected. :-)

no worries, the FS idea sounded interesting enough to knock out a quick
client. But given the UI needed, it won't be a very useful client without a
lot more work so we might as well go the full length there.
 
> I'm sorely tempted to buy one of these things to hack on it to be
> honest.

if you do, buy a folio. we don't know if that works yet :)

> If you're producing svgs as the end of your process, then you can have
> inkscape open them. Accommodations can be made if we want Inkscape to
> be aware of any new attributes you add in to retain useful data.

our file format atm is just a json file with the raw data (x/y/pressure in
arrays that are the strokes). we can produce whatever we want from that. I
hear the mp3 patent has expired ;)

Cheers,
   Peter


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel
Reply | Threaded
Open this post in threaded view
|

Re: Wacom smartpad support - request for ideas

Martin Owens-2

> > I'm sorely tempted to buy one of these things to hack on it to be
> > honest.
> if you do, buy a folio. we don't know if that works yet :)

OK I got the small folio, got a friendly discount at B&H so only $120.

> our file format atm is just a json file with the raw data
> (x/y/pressure in
> arrays that are the strokes). we can produce whatever we want from
> that. I
> hear the mp3 patent has expired ;)

Unfortunatly inkscape doesn't support mp3s ;-P

Martin,

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Inkscape-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/inkscape-devel