QStandardPaths::GenericDataLocation on MacOS

[Thomas Friedrichsmeier]

Hi!

Quick summary, first:
KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
that location is problematic on Mac because it refers to absolute,
non-customizable system paths, only. We'd like to avoid patching Qt, so
we are looking to come up with a solution that has a chance of getting
upstreamed.

The problem in a lot more detail:
Feel free to skip over any sections that you are already familiar with.

-- Usage of GenericDataLocation in KF5 --
Many KF5 frameworks store data or look for data in
QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
are just two such examples, which, importantly, are designed such that
users (people or applications) can supply additional data files in
specific directories, such as returned by
QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
QStringLiteral("kxmlgui5/"))
Using QStandardPaths::GenericDataLocation has also been suggested
in places such as
https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .

Nowadays compiled in qrc resources are used instead of installed files,
increasingly, and this removes any worries about file system paths.
However, due to the above we should assume that use of
QStandardPaths::GenericDataLocation is wide-spread in KF5-based
applications, and must remain supported.

-- The situation on Linux --
On Unix systems (not Mac), the paths returned for
QStandardPaths::GenericDataLocation correspond to those of the
XDG_DATA_DIRS environment variable. These will usually be system wide
paths including /usr/share and /usr/local/share. Since the typical
application deployment happens to one of those directories, installed
files will be accessible to frameworks out of the box. In case of
non-standard installation paths, modifying XDG_DATA_DIRS provides a
straight-forward mechanism to adjust lookup.

-- The situation on Windows --
On Windows, GenericDataLocation is mapped to the system specific
"FOLDERID_LocalAppData", which includes some system wide paths, but
also "<APPDIR>", and "<APPDIR>/data", where <APPDIR> is the location of
the binary. While these paths are not directly controllable, having the
relative path in them allows
- installing a group of applications sharing the same file or
- installing a single application (with supporting frameworks)
both to arbitrary installation paths. The only thing that is not -
easily - possible compared to Unix, is having two Applications
installed to two separate installation root directories, and still
sharing the same data files.

The expected filesystem layout is somewhat different from Unix (data
instead of share), but those differences can be handled (and are
handled) by ECM:
https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstallDirs.cmake#L529
. ("<APPDIR>/data" is the path chosen from the available options).

-- The situation on MacOS --
On MacOS, GenericDataLocation is mapped to the system specific
"NSApplicationSupportDirectory", which contains the paths
"/Library/Application Support/" and "~/Library/Application Support/".
Notably, any installation-relative path is missing, as are any options
for customization.

As an interesting aside, the handling is different for AppDataLocation
and AppLocalDataLocation, which add the relative path
"<APPDIR>/../Resources/" (and also append "<APPNAME>" to all paths).
This matches with the notion that data either belongs to exactly one
application (AppDataLocation) or to _all_ applications on the system
(GenericDataLocation), but it does not support the notion that there
may be a group/bundle of applications sharing their data, or that a
single application may look up installation-relative data in any other
than "<APPDIR>/../Resources/<APPNAME>". (Importantly not in
"<APPDIR>/../Resources/kxmlgui5", for intance).

We'll talk about this in a bit, but for now, back to the
current definition on GenericDataLocation on Mac:

This setup severely limits the viable deployment options for KF5-based
applications on Mac to basically just one: They must be installed (or
at least their data must be installed) to a fixed central system
folder. Unfortunately, this conflicts with all major existing
approaches:

- MacPorts: Uses a centralized installation, _but_ inside an own
installation root (/opt/local/ by default), and is designed not to
touch any paths outside that installation root. MacPorts _still_ does
not have any official KF5 packages, and the ones provided by our own
kde-mac people rely on a patched Qt (details in a bit).
- Homebrew: Also a centralized installation, also inside its own
installation root (typically in a user's home directory). Can cope
better because it is not afraid to create symlinks outside its
installation root (we'll discuss that, too).
- Craft ("normal installation"): Again, a centralized installation,
typically inside a user's home directory. No real solution, to the
problem, yet (the reason for this mail).
- Craft ("application bundle images"): Craft can also create .dmg
packages from single applications (and their dependencies), which
provides for very easy drag-and-drop application installers (much
like a .tar.gz or .zip). Right now these support installation anywhere
(system or user folders), without the need to touch anything outside
that single installation directory. A feature that would certainly be
nice to keep. No solution to the GenericDataLocation-problem, yet.

-- Existing approaches to the problem --
--- Symlinks ---
As a solution that does not require any changes to Qt, an installation
routine could place symlinks to the real installation directories
in /Library/Application Support/ or ~/Library/Application Support/ .
However this approach has some important drawbacks:
1. It requires "smart" installers and uninstallers, will not work with
plain .dmg's.
2. Once installed, an application cannot trivially be moved, any more,
as that would also require adjusting the symlinks.
3. It only really works for a single centralized installation. However,
if installing two KF5-based applications AppA and AppB to separate
installation roots, where would - for instance - ~/Library/Application
Support/kxmlgui5 point to? A directory inside AppA's installation
root, or inside AppB's installation root? It cannot be both.

--- Patching in Unix-like paths ---
The "kde-mac/Macports" people have created a patch to QStandardPaths
that will essentially enable unix-like paths on request of the app. The
most recent version is this (is believe):
https://github.com/RJVB/macstrop/blob/master/aqua/qt5-kde/files/qt593/fix-qstandardpaths6.patch
A stale Qt-Ticket is here: https://bugreports.qt.io/browse/QTBUG-44473 .

I do not want to (and am not qualified to) go into the details of this
rather large patch, but I think for the purpose of this discussion, the
key idea of the approach can be summarized as "include the
XDG_DATA_DIRS in GenericDataLocation paths, as on Unix".

The advantage of this approach is that it provides a mechanism for the
using application to adjust QStandardPaths-behavior to whatever is
appropriate. The drawbacks I can think of:
1. A rather large patch
2. There may be a concern about side-effects, as the behavior of Qt
will depend on an environment variable where it did not before, with
_potential_ interaction between separate installations.
3. Will also need some "cooperation" from the calling app in order to
work for all installation scenarios (XDG_DATA_DIRS will have to be
set/adjusted, and an extra setup call may be needed to request
"Unix-behavior" from QStandardPaths).

--- Patching in an installation-relative path ---
A much smaller patch
(https://cgit.kde.org/craft-blueprints-kde.git/tree/libs/qt5/qtbase/fix_GenericDataLocation_mac.patch)
could be used to add "<APPDIR>/../Resources" to the end of the path
list returned for GenericDataLocation. This would allow to locate
resources in any installation root (be it centralized or single-app),
albeit in a non-configurable relative path inside the installation
root. This would thus need support from ECM to adjust the
installation-layout, accordingly, _or_ the addition of a symlink
- _inside_ the installation root, this time - e.g. from ROOT/Resources
to ROOT/share .

Not sure about the MacPorts-take on doing something like this, but for
Homebrew and Craft (both setups), this should(TM) be no problem. As the
new search directory is added at the end of the path list, _and_ it is
at an installation-relative path (where only files belonging to that
installation will be found), the potential for unexpected side-effects
should be strictly limited.

As one drawback I can think of, handling supporting binaries, e.g. in
"libexec" may cause a headache, although this, too should probably be
solvable by appropriate adjustments of the installation layout or
symlinks.

-- Other ideas --
Another idea would be to allow qt.conf to influence the paths returned
for GenericDataLocation. As qt.conf can be placed next to the
application to deploy, lookup behavior could be adjusted in a
straight-forward way, limited to each installation.

As, so far, qt.conf does not affect QStandardPaths at all, however,
such a patch would probably be rather intrusive in itself. (Also, so
far, qt.conf only support one path per type, not a list of paths).

-- Other things to consider --
Essentially the same thoughts probably apply to GenericConfigLocation.
I left that out for now, in order to not make the discussion even more
complex. And of course the solution should be transferrable to further
platforms beyond Unix/Windows/Mac, if needed.

-- The bottom line --
Ok, sorry for the long mail. Long story short: We need to come up with
a solution, and it best be a solution that
a) can be upstreamed to Qt
b) works out for MacPorts, Homebrew, _and_ Craft.
So does any of the approaches outlined above qualify for these
conditions, or what else could be a viable route to take?

Regards
Thomas

[René J.V. Bertin]

On Thursday May 03 2018 10:02:35 Thomas Friedrichsmeier wrote:

Hi,

I have mostly skimmed over your long email but allow me to warn you firsthand that I've tried to get feedback and constructive collaboration like this before and never got very far because the prevailing attitude is 1) don't patch Qt and 2) be as Mac'ish as possible (without necessarily having a lot of experience with what that actually means).

What is often overlooked is that not all of Apple's own (and 3rd-party) software installs via simple DMGs, notably applications that depend on or provide shared resources. I would assume that iTunes still installs via an installer, for instance.

There are individual project efforts that manage to get a working standalone app bundle using an unpatched Qt and QSP. It is rather evident that this should be possible because Qt designed QSP for this purpose AND to allow using Qt for apps offered via Apple's app store. This works well enough for applications that don't don't depend on others and don't need DBus to advertise their services.

But in the end it's the KDE model itself that cannot easily (if at all) be adapted to desktop systems like Macs and MSWin PCs if you do not want to provide a central install of the shared resources including certain external dependencies like DBus. On Mac the "native" way to do that would be a single framework in /Library/Frameworks which bundles all KF5 frameworks like certain of Apple's frameworks contain multiple sub-frameworks, plus a corresponding store in /Library/Application Support where other KF5 libraries, plugins etc. can install their support files.
That would of course represent a huge effort that should have been started immediately with KF5 5.0.0 .

A couple of years into "supporting" KF5 on Mac and trying to get traction for it on/in MacPorts I have come to the conclusion that there just is no demand beyond possibly the few applications that are sufficiently self-sufficient to be offered "as an icon on a DMG"; Kate, probably KDevelop, Krita and possibly the other Calligra members and maybe a handful of others. digiKam has a Mac installer, apparently its target audience is expected to have enough computer literacy to know what to do with one of those :)

About MacPorts and official KF5 ports: unlikely to happen unless someone else is willing to recreate ports so you get a sensible experience with the unpatched Qt5 provided by the official Qt5 port (or adapt my KF5 ports minimally to that effect). The hurdle here is that I've never been able to arrive at a constructive collaboration with the Qt5 port maintainer; without that there will be no official alternative Qt5 port for use with KDE and we'll remain stuck with the need for a custom ports tree.

Post by Thomas Friedrichsmeier
1. It requires "smart" installers and uninstallers, will not work with
plain .dmg's.
2. Once installed, an application cannot trivially be moved, any more,
as that would also require adjusting the symlinks.
3. It only really works for a single centralized installation. However,
if installing two KF5-based applications AppA and AppB to separate
installation roots, where would - for instance - ~/Library/Application
Support/kxmlgui5 point to? A directory inside AppA's installation
root, or inside AppB's installation root? It cannot be both.
1. A rather large patch

It is largish because designed with upstreamability in mind and I thin it also still contains echoes of earlier approaches that could be removed. A patch for a Qt version only to be used with KF5 code can be a lot leaner.
The stale ticket you refer to is probably the one on which it was said that the Qt devs are open for this kind of patch, provided it is well justified (argued) and it is transparent by default so that AppStore compatibility isn't jeopardised. My idea has always been to test and tune my patch in the wild and then present it again with community back-up.
There has also been an exchange with someone from the MSWin universe about linking this patch to to-be-developed qt.conf functionality, e.g. to replace the XDG_DATA_DIRS env. variable or possibly even the option to define all (alternative) QSP locations via qt.conf .

Post by Thomas Friedrichsmeier
2. There may be a concern about side-effects, as the behavior of Qt
will depend on an environment variable where it did not before, with
_potential_ interaction between separate installations.

The env. variable is not required, it was added to conform with the behaviour on other Unix variants but I never use it myself.
XDG_DATA_DIRS is not an env. variable that is used by other Mac software, but it shouldn't be that hard to find an alternative for it should that be necessary.

Post by Thomas Friedrichsmeier
3. Will also need some "cooperation" from the calling app in order to
work for all installation scenarios (XDG_DATA_DIRS will have to be
set/adjusted, and an extra setup call may be needed to request
"Unix-behavior" from QStandardPaths).

No, you only need to set a preprocessor token or two during compilation. That is something that could easily be handled via the ECM.

Post by Thomas Friedrichsmeier
Not sure about the MacPorts-take on doing something like this, but for

MacPorts (and Fink, which still exists!) use a single prefix in which everything is installed like Gentoo Prefix or pkgsrc would do it. The goal is to provide the best and easiest way to run generic Unix applications as if you were working on a Unix workstation (wait, Macs actually are Unix workstations, too).
This includes interoperability with GTk/Gnome applications and anything else that follows the XDG/FreeDesktop standards and thus need to find certain resources in well-defined locations if they are to know about KF5 applications, and vice-versa(e.g. : open an SVG in Inkscape from a KF5 app or a text file in Kate from a GTk/Gnome app).

Post by Thomas Friedrichsmeier
As one drawback I can think of, handling supporting binaries, e.g. in
"libexec" may cause a headache, although this, too should probably be
solvable by appropriate adjustments of the installation layout or
symlinks.

"libexec" and the Qt5/KF5 plugins directory could go into /Library/Application Support/KF5 for instance. I'd strongly advise against symlinks, unless they're all pointing to central locations. And not for libraries; the Mac linker will store normalised paths to shared libraries.

Post by Thomas Friedrichsmeier
As, so far, qt.conf does not affect QStandardPaths at all, however,
such a patch would probably be rather intrusive in itself. (Also, so
far, qt.conf only support one path per type, not a list of paths).

Patch size is not a problem at least if the patch is potentially upstreamable. The problem I see with qt.conf is that it is application-specific; each app bundle will have to provide its own copy.

Post by Thomas Friedrichsmeier
Ok, sorry for the long mail. Long story short: We need to come up with
a solution, and it best be a solution that
a) can be upstreamed to Qt
b) works out for MacPorts, Homebrew, _and_ Craft.
So does any of the approaches outlined above qualify for these
conditions, or what else could be a viable route to take?

I've had enough time to muse over this to be sufficiently certain that my QSP patch would be the way to go if we're talking about upstreaming a patch to that class.

The alternative would be to convince Christoph Cullmann and like-minded people to work on a way to mould their respective approaches for building standalone app bundles into something that can be added to the ECM. In my eyes that could lead ultimately to a CMake build option that will enable a standalone app bundle build when building on Mac (or disable it if the default though that should probably be project-specific; I believe something like KDE PIM cannot be bundled that way for instance).
I'd be willing to participate in such an effort though mostly to guarantee that "linuxy builds" will continue to work because I have no interest in or use for standalone bundled versions myself.

As to "our own kde-mac" people: nowadays that seems to be just me ... and it's very likely that I will drop out too when my current Mac dies or becomes too slow.

R.

[Thomas Friedrichsmeier]

Hi,

I'll just answer a few select points, because, yes I am aware this has
a tiny bit of a history, and I don't want to see a repeat. The master
plan is actually to get David Faure to champion a solution (whichever
one) into Qt. So before getting lost in the detail of the various
approaches, I'd like to hear a word on the general direction that seems
most promising.

On Thu, 03 May 2018 11:24:51 +0200
René J.V. Bertin <***@gmail.com> wrote:
[...]

Post by René J.V. Bertin
What is often overlooked is that not all of Apple's own (and
3rd-party) software installs via simple DMGs, notably applications
that depend on or provide shared resources. I would assume that
iTunes still installs via an installer, for instance.

True, yet another route would be to simply require KF5-based apps to
install to /Library/Application Support. But that solution has obvious
limitations that I would like to avoid, and - by analogy to Unix
_and_ Windows - are entirely avoidable.

Post by René J.V. Bertin
There are individual project efforts that manage to get a working
standalone app bundle using an unpatched Qt and QSP. [...]

It's not even hard for individual apps, today. Just compile in all
data files as qresources, and the problem is done with. As an
application maintainer that is my fallback plan. But it's not really my
preferred vision for KF5 as a whole.

Post by René J.V. Bertin
But in the end it's the KDE model itself that cannot easily (if at
all) be adapted to desktop systems like Macs and MSWin PCs if you do
not want to provide a central install of the shared resources
including certain external dependencies like DBus.

Yes, but "central" is pretty relative, here. I know single app
installers are not your vision of KF5 on Mac, and I certainly don't
want to push them as _the_ way to go, but to be honest, I'm quite
impressed with the .dmg's created on the binary-factory for Mac, today.
A fully packaged Kate is available at 88MB download for instance,
despite including a whole bunch of frameworks.

Yet that seems mostly orthogonal to the issue at hand to me. Both
monolithic and single app installation are facing the same problem
right now, and essentially need (one of) the same solution to be able
to fix it.

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
Not sure about the MacPorts-take on doing something like this, but for

MacPorts (and Fink, which still exists!) use a single prefix in which
everything is installed like Gentoo Prefix or pkgsrc would do it.

I'm absolutely aware of that. The question is simply: Would MacPorts
tolaterate a symlink from /opt/local/Resources to /opt/local/share
(/opt/local being the installation prefix)? If so, the simple patch I
presented should immediately allow a clean way forward for KF5 on
MacPorts (if upstreamed).

Post by René J.V. Bertin
Patch size is not a problem at least if the patch is potentially
upstreamable.

But it could be, if the responsible reviewer is a bit uninterested...

Post by René J.V. Bertin
The problem I see with qt.conf is that it is
application-specific; each app bundle will have to provide its own
copy.

True, but so what, if it can be generated, automatically.

Post by René J.V. Bertin
The alternative would be to convince Christoph Cullmann and
like-minded people to work on a way to mould their respective
approaches for building standalone app bundles into something that
can be added to the ECM.

Again, I think this is not an either-or situation. Both centralized and
standalone app installations are affected by this problem, unless
relying on .qrc files, exclusively.

Regards
Thomas

[René J.V. Bertin]

On Thursday May 03 2018 15:19:33 Thomas Friedrichsmeier wrote:

Hi,

Post by Thomas Friedrichsmeier
plan is actually to get David Faure to champion a solution (whichever
one) into Qt.

I can be wrong, but I don't see that happening beyond his general support.
Also, what are the chances that this will be overhauled anyway for Qt6?

Post by Thomas Friedrichsmeier
install to /Library/Application Support. But that solution has obvious
limitations that I would like to avoid, and - by analogy to Unix
_and_ Windows - are entirely avoidable.

I don't see what Unix does here. /Library/Application Support is just the OS X equivalent of $prefix/share and maybe parts of $prefix/libexec; both are central, shared location that have a per-user equivalent (~/Library/Application Support and ~/.local/share). The limitations I see are purely and only related to the way software has to be installed.

Post by Thomas Friedrichsmeier
It's not even hard for individual apps, today. Just compile in all
data files as qresources, and the problem is done with.

That may be true for resources that contain no executable native code, but it probably won't work for libraries and plugins.

Post by Thomas Friedrichsmeier
A fully packaged Kate is available at 88MB download for instance,
despite including a whole bunch of frameworks.

But with that you get a version of the application that just drops the functionality it gets through DBus, uses the alien-looking Breeze icons and default colour palette, plus the "native" Mac widget style which makes it look like an application for the visually impaired (everything way too big). It works, but IMHO no longer has any advantages over well-established truly native competitors like BBEdit (which clocks in at 32Mb *uncompressed*).
And: supposing those 88Mb are the uncompressed app bundle size, how much of that has to be duplicated in each and every app that uses the Kate kpart?

Post by Thomas Friedrichsmeier
I'm absolutely aware of that. The question is simply: Would MacPorts
tolaterate a symlink from /opt/local/Resources to /opt/local/share

They probably wouldn't like it, but if the dislike is strong enough it will lead to a small patch that undoes or modifies the change.
Careful though, <APPDIR> should probably be something like QLibraryInfo::prefixPath and not the method from QApplication which will point to the MacOS directory inside the app bundle.

Post by Thomas Friedrichsmeier
Again, I think this is not an either-or situation. Both centralized and
standalone app installations are affected by this problem, unless
relying on .qrc files, exclusively.

There *is* an either-or situation: work with the existing QStandardPaths or don't. Kate.app shows that the former is not impossible, and it has the big advantage that you do everything in-house without modifying Qt.

Another thing: it seems that in your reasoning you have only considered the search algorithms for finding readable locations. For the writable locations you cannot simply append a "compatibility" location, and in turn that means that resources generated at runtime will later be found in "native" Mac locations because QSP will return the first location that has the requested file.

Which reminds me that I've already suggested several times that it would be useful to rethink the build system so that it determines the install locations based on QSP (using qtpaths or a dedicated small utility).

Cheers,
R.

[Thomas Friedrichsmeier]

Hi,

On Thu, 03 May 2018 16:01:46 +0200

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
plan is actually to get David Faure to champion a solution (whichever
one) into Qt.

I can be wrong, but I don't see that happening beyond his general support.

well, let's hear him on this.

Post by René J.V. Bertin
I don't see what Unix does here. /Library/Application Support is just
the OS X equivalent of $prefix/share and maybe parts of
$prefix/libexec; both are central, shared location that have a
per-user equivalent (~/Library/Application Support and
~/.local/share). The limitations I see are purely and only related to
the way software has to be installed.

Yes, it's about the installation options. It is _primarily_ about the
fact that the one theorically possible installation option on Mac
matches with _none_ of the major available solutions (macports,
homebrew, craft). It is also about the fact that this installation
option causes problems, e.g. if users are to mix MacPorts and a
single-app-bundled KF5 app, or two versions of a KF5 app, or many other
not-so-unlikely scenarios.

Unix does support installing software to any prefix
(because it provides a way to customize lookup). Windows supports
installing software to any prefix (because it includes an
installation-relative path in the list of paths for
GenericDataLocation). Mac does not, because it lacks both (even though
installation-relative paths are available in other QStandardPaths).

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
It's not even hard for individual apps, today. Just compile in all
data files as qresources, and the problem is done with.

That may be true for resources that contain no executable native
code, but it probably won't work for libraries and plugins.

Those can be handled in qt.conf (and macdeployqt auto-generates an
appropriate qt.conf, for instance). That may or may not be our favorite
solution, but it is an already available option (and if there is a
concern that this could cause problems for monolithic (macports-style)
installations, qt.conf can even be provided as a per-app qresource,
which in principle, should be possible to generate, and compile into
each binary, automatically.

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
A fully packaged Kate is available at 88MB download for instance,
despite including a whole bunch of frameworks.

But with that you get a version of the application that just drops
the functionality [...]

I just don't want to get into that discussion, at all, here. I
absolutely see use-cases for both "monolithic" and "single app"
installations. So both should be supported. Currently neither is. So
the issue at hand isn't to discuss which of the two is superior, but to
find out how to fix the problem affecting them both.

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
Again, I think this is not an either-or situation. Both centralized
and standalone app installations are affected by this problem, unless
relying on .qrc files, exclusively.

There *is* an either-or situation: work with the existing
QStandardPaths or don't. Kate.app shows that the former is not
impossible, and it has the big advantage that you do everything
in-house without modifying Qt.

QRC, sing it with me. That's the whole secret, as far as I am aware:
Circumventing the problem by not installing files. Since I found out
about this, I sleep a lot better as maintainer of RKWard, but it still
bugs me, severly, for KF5 as a whole.

Post by René J.V. Bertin
Another thing: it seems that in your reasoning you have only
considered the search algorithms for finding readable locations. For
the writable locations you cannot simply append a "compatibility"
location, and in turn that means that resources generated at runtime
will later be found in "native" Mac locations because QSP will return
the first location that has the requested file.

Why would that be a problem? If something is written to the writable
location, it will then later be found. Sure, that's the point, isn't it.
Possibly by a different application, or by a different version of the
same application. Also true, but nothing new.

Regards
Thomas

[Ben Cooksley]

On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier

Post by Thomas Friedrichsmeier
Hi,
On Thu, 03 May 2018 16:01:46 +0200

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
plan is actually to get David Faure to champion a solution (whichever
one) into Qt.

I can be wrong, but I don't see that happening beyond his general support.

well, let's hear him on this.

Post by René J.V. Bertin
I don't see what Unix does here. /Library/Application Support is just
the OS X equivalent of $prefix/share and maybe parts of
$prefix/libexec; both are central, shared location that have a
per-user equivalent (~/Library/Application Support and
~/.local/share). The limitations I see are purely and only related to
the way software has to be installed.

Yes, it's about the installation options. It is _primarily_ about the
fact that the one theorically possible installation option on Mac
matches with _none_ of the major available solutions (macports,
homebrew, craft). It is also about the fact that this installation
option causes problems, e.g. if users are to mix MacPorts and a
single-app-bundled KF5 app, or two versions of a KF5 app, or many other
not-so-unlikely scenarios.
Unix does support installing software to any prefix
(because it provides a way to customize lookup). Windows supports
installing software to any prefix (because it includes an
installation-relative path in the list of paths for
GenericDataLocation). Mac does not, because it lacks both (even though
installation-relative paths are available in other QStandardPaths).

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
It's not even hard for individual apps, today. Just compile in all
data files as qresources, and the problem is done with.

That may be true for resources that contain no executable native
code, but it probably won't work for libraries and plugins.

Those can be handled in qt.conf (and macdeployqt auto-generates an
appropriate qt.conf, for instance). That may or may not be our favorite
solution, but it is an already available option (and if there is a
concern that this could cause problems for monolithic (macports-style)
installations, qt.conf can even be provided as a per-app qresource,
which in principle, should be possible to generate, and compile into
each binary, automatically.

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
A fully packaged Kate is available at 88MB download for instance,
despite including a whole bunch of frameworks.

But with that you get a version of the application that just drops
the functionality [...]

I just don't want to get into that discussion, at all, here. I
absolutely see use-cases for both "monolithic" and "single app"
installations. So both should be supported. Currently neither is. So
the issue at hand isn't to discuss which of the two is superior, but to
find out how to fix the problem affecting them both.

It is worth noting that at one point we had a much lighter (only about
20MB or so I think) DMG however that broke icon support.
There is definitely space for optimisation here in terms of what we're
including in the .app bundle.

If we end up going down the QRC route (which also has the advantage of
improving speed as it eliminates the large number of small files) then
we can also ensure the icons are kept compressed on disk.

Post by Thomas Friedrichsmeier

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
Again, I think this is not an either-or situation. Both centralized
and standalone app installations are affected by this problem, unless
relying on .qrc files, exclusively.

There *is* an either-or situation: work with the existing
QStandardPaths or don't. Kate.app shows that the former is not
impossible, and it has the big advantage that you do everything
in-house without modifying Qt.

Circumventing the problem by not installing files. Since I found out
about this, I sleep a lot better as maintainer of RKWard, but it still
bugs me, severly, for KF5 as a whole.

Post by René J.V. Bertin
Another thing: it seems that in your reasoning you have only
considered the search algorithms for finding readable locations. For
the writable locations you cannot simply append a "compatibility"
location, and in turn that means that resources generated at runtime
will later be found in "native" Mac locations because QSP will return
the first location that has the requested file.

Why would that be a problem? If something is written to the writable
location, it will then later be found. Sure, that's the point, isn't it.
Possibly by a different application, or by a different version of the
same application. Also true, but nothing new.
Regards
Thomas

Cheers,
Ben

[René J.V. Bertin]

Post by Ben Cooksley
On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier

Post by Thomas Friedrichsmeier
homebrew, craft). It is also about the fact that this installation
option causes problems, e.g. if users are to mix MacPorts and a
single-app-bundled KF5 app

No, that shouldn't happen, and it doesn't with the official Kate bundle. Standalone & self-sufficient also implies that you don't clash with another similar application, or even with another copy of yourself.
Bundling DBus within such a standalone app bundle and running it from there would of course be asking for trouble, but it would also make no sense at all.

Post by Ben Cooksley

Post by Thomas Friedrichsmeier
Unix does support installing software to any prefix
(because it provides a way to customize lookup).

More importantly, the Unix QSP are relative to Qt's install prefix (if memory serves me well; and my patched version is too).

Post by Ben Cooksley

Post by Thomas Friedrichsmeier
concern that this could cause problems for monolithic (macports-style)

I think you're misusing the term monolithic here? Standalone appbundles are monolithic, an aggregate framework bundle containing all KF5 frameworks would be monolithic too. Installing under a prefix that isn't the root is not what I'd call monolithic (and in practice MacPorts installs its .app bundles under /Applications, not under $prefix ;))

Post by Ben Cooksley
improving speed as it eliminates the large number of small files) then
we can also ensure the icons are kept compressed on disk.

Doesn't Kate already get its icons from the qresource?

Post by Ben Cooksley

Post by Thomas Friedrichsmeier
Why would that be a problem? If something is written to the writable

It's not a problem if you don't care where those files are written. But if that's true, the whole issue can probably be avoided by determining the install locations for (shared) resources from QSP instead of hardcoding them in CMake files...

1 problem here: ApplicationsLocation. On Mac that points to /Applications (probably still without a test version - danger!), which is not at all comparable to what ApplicationsLocation is used for elsewhere.

Cheers,
R.

[Thomas Friedrichsmeier]

On Thu, 03 May 2018 21:09:03 +0200

Post by René J.V. Bertin

Post by Ben Cooksley
On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier

Post by Thomas Friedrichsmeier
homebrew, craft). It is also about the fact that this installation
option causes problems, e.g. if users are to mix MacPorts and a
single-app-bundled KF5 app

No, that shouldn't happen, and it doesn't with the official Kate
bundle. Standalone & self-sufficient also implies that you don't
clash with another similar application, or even with another copy of
yourself. Bundling DBus within such a standalone app bundle and
running it from there would of course be asking for trouble, but it
would also make no sense at all.

Again, this is about QStandardPaths::GenericDataLocation, exclusively.
_If_ applications were to actually install to the current paths defined
for that on Mac "Library/Application Support", and "~Library/Application
Support", on more, no less, _then_ clashes would start to happen. As
far as I am aware, _none_ of the current installation approaches for
KF5 does so, and as a consequence they all fail GenericDataLocation
lookup with an unpatched qt.

Which is one reason why I don't think "just install to the offical path,
then" is an acceptable solution.

Current kate "standalone" is not a good example, because it is
_circumventing_ any QStandardPaths::GenericDataLocation lookup problems
by shipping all its data compiled in. If it did not, it would fail to
find its data files with an unpatched qt, despite following documented
code patterns.

Post by René J.V. Bertin
I think you're misusing the term monolithic here? Standalone
appbundles are monolithic, an aggregate framework bundle containing
all KF5 frameworks would be monolithic too. Installing under a prefix
that isn't the root is not what I'd call monolithic

I don't care what it's called, I just need some terms to refer to this.
My monolithic, here, was referring to _all_ in one place, instead of
one app and its dependencies in one place. And yes, standalone bundles
are not really all that different from "monolithic" installs,
technically, except, it would be a typical situation to have more than
one of them at the same time.

Post by René J.V. Bertin
(and in practice
MacPorts installs its .app bundles under /Applications, not under
$prefix ;))

Ah, good reminder. But it doesn't install anything under /Library, does
it? Which is why MacPorts, too, cannot support KF5 with a non-patched
QStandardPaths, right?

Post by René J.V. Bertin

Post by Ben Cooksley

Post by Thomas Friedrichsmeier
Why would that be a problem? If something is written to the
writable

It's not a problem if you don't care where those files are written.

I don't care where files are written at runtime (as long as they can be
found again). I would like not to care about where files are written at
installation time, but the sad fact is they will not be found unless
installed to one specific absolute path.

Post by René J.V. Bertin
But if that's true, the whole issue can probably be avoided by
determining the install locations for (shared) resources from QSP
instead of hardcoding them in CMake files...

Nope. That will simply lead us into the situation discussed near the top
of this mail. Besides, _if_ we are both talking about the option of
using an _unpatched_ Qt, then hardcoding or querying QSP makes very
little practical difference.

I understand you'd like to see ECM to continue to support your
"Unix-like" patched QSP, and why not? ECM is under "our" control
anyway, but the issue at hand is one that cannot be solved in ECM alone.

Post by René J.V. Bertin
1 problem here: ApplicationsLocation. On Mac that points
to /Applications (probably still without a test version - danger!),
which is not at all comparable to what ApplicationsLocation is used
for elsewhere.

True. Any code that relies on this will likely fail on Mac _and_
Windows, so that's a valid concern. Yet, arguably, its typically going
to be a milder failure, and clearly, the prevalence is much lower
(check GneericDataLocation vs. ApplicationsLocation on lxr.kde.org). So
let's keep it in mind, but not make it a key concern while looking for
a path forward. (And in fact, I'm a bit at a loss to see how to solve
_that_, short of, again, installing everything to a single absolute
location.)

Regards
Thomas

[René J.V. Bertin]

Post by Thomas Friedrichsmeier
_If_ applications were to actually install to the current paths defined
for that on Mac "Library/Application Support", and "~Library/Application
Support", on more, no less, _then_ clashes would start to happen. As

A priori no, because the Application Support directory shouldn't be written to directly (as per Apple's rules) but should contain subdirectories, e.g. /Library/Application Support/Kate . I've never really looked at how that works in practice, but I strongly doubt that code has to append this subdirectory itself, that'd be contrary to the QSP purpose.

Post by Thomas Friedrichsmeier
far as I am aware, _none_ of the current installation approaches for
KF5 does so, and as a consequence they all fail GenericDataLocation
lookup with an unpatched qt.

Nope; my hope is that this would remove the need for patching QSP. As you said yourself, files written to a QSP location at runtime can be found again through QSP. That should apply to files written during the install step too (as if the application writes them to that location during its first run).

Post by Thomas Friedrichsmeier
I don't care what it's called

Still, best to avoid terms that are confusing because others interpret them differently.

Post by Thomas Friedrichsmeier
Ah, good reminder. But it doesn't install anything under /Library, does
it? Which is why MacPorts, too, cannot support KF5 with a non-patched
QStandardPaths, right?

Good question, actually, because this doesn't apply only to KF5 code. It's not forbidden to install things outside $prefix, but it's discouraged. I'm not aware of any "pure Qt5" application which installs things into /Library/Application Support

Post by Thomas Friedrichsmeier
of this mail. Besides, _if_ we are both talking about the option of
using an _unpatched_ Qt, then hardcoding or querying QSP makes very
little practical difference.

One very important difference: querying should work on all platforms, so remove the need for conditional code.

Post by Thomas Friedrichsmeier
I understand you'd like to see ECM to continue to support your
"Unix-like" patched QSP, and why not? ECM is under "our" control
anyway, but the issue at hand is one that cannot be solved in ECM alone.

Forgive me if I'm not entirely reassured that there won't be CMake code at some point that simply checks `if(APPLE)` to decide whether or not to use tweaks for generating standalone app bundles (Marble does, and its devs have simply ignored my proposed patch to reenable "linuxy" builds).

Post by Thomas Friedrichsmeier
Windows, so that's a valid concern. Yet, arguably, its typically going
to be a milder failure, and clearly, the prevalence is much lower

Until you run the unittests and one of those just trashes ApplicationsLocation when it's done. That actually happened to me, and I still can't believe I actually caught that before my entire /Applications had disappeared.

Post by Thomas Friedrichsmeier
a path forward. (And in fact, I'm a bit at a loss to see how to solve
_that_, short of, again, installing everything to a single absolute
location.)

The usual content of ApplicationsLocation (.desktop files) probably makes no sense anyway for standalone app bundles that are not expected to play nice with applications following FreeDesktop protocols. IOW, those files could be skipped during the installation step in standalone build mode (again, not simply `if(APPLE)`).

Anyway, I don't want to monopolise this thread and I'm beginning to feel that's exactly what I'm doing. Apologies.

R.

[Thomas Friedrichsmeier]

I'll shuffle the bits in this mail a bit, in an attempt to get the
discussion back to the main topic (or at least the thing that I
intended to be the main topic):

On Thu, 03 May 2018 23:11:56 +0200

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
Ah, good reminder. But it doesn't install anything under /Library,
does it? Which is why MacPorts, too, cannot support KF5 with a
non-patched QStandardPaths, right?

Good question, actually, because this doesn't apply only to KF5 code.
It's not forbidden to install things outside $prefix, but it's
discouraged. I'm not aware of any "pure Qt5" application which
installs things into /Library/Application Support

Exactly. And that returns us to Square 1: Using GenericDataLocation
simply does not work for us on Mac, but KF5 apps use it all over the
place (encouraged by documentation). We will either have to stop using
it, today, or patch Qt (upstream / downstream).

So where do we go from here?






^This space intentionally left blank to emphasize the above.^

Post by René J.V. Bertin
Nope; my hope is that this would remove the need for patching QSP. As
you said yourself, files written to a QSP location at runtime can be
found again through QSP. That should apply to files written during
the install step too (as if the application writes them to that
location during its first run).

Ok, so you're talking about _installing_ to the writeable location.
Would probably work, but
1. It breaks the current separation between "vendor provided" and "user
added / customized", including the implicit protection of user files
that this provides.
2. It still does not allow for an installation to be limited to a
single arbitrary prefix, with the benefits that provides (multiple
versions / methods of installation for one application, trivial
uninstallation, easy archiving, running from thumb drives, etc.)

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
I don't care what it's called

Still, best to avoid terms that are confusing because others
interpret them differently.

Well, sorry for causing confusion, then. So we agree on calling the one
thing "standalone apps (bundles)", but what is the complementary term?

[...]

Post by René J.V. Bertin
The usual content of ApplicationsLocation (.desktop files) probably
makes no sense anyway for standalone app bundles that are not
expected to play nice with applications following FreeDesktop
protocols. IOW, those files could be skipped during the installation
step in standalone build mode (again, not simply `if(APPLE)`).

Besides any code querying ApplicationsLocation is _probably_ broken at
least on Windows, irrespective of the filesystem location. Which does
not turn this into a non-issue, but into a separate issue, that IMO
should be discussed separately (and probably requires individual
fixes, instead of and "global" measure).

Regards
Thomas

[René J.V. Bertin]

Hi,

I just remembered that some time ago I wrote a small library that allows me to use binaries compiled against my own patched qt5-kde with an "official" Qt build: it provides the additional symbols that are in qt5-kde but not standard Qt through code injection (DYLD_INSERT_LIBRARIES). That isn't really a kosher thing to do (and actively discouraged on newer Mac OS versions) but it brings me to this:

Could we design a kind of Qt plugin that provides an alternative QSP, a bit like how the plasma-integration plugin provides the KDE file dialog? In first testing/internal approach that could use a similar trick as the one I'm using with my QExtStandardPaths (a macro definition given in CPPFLAGS) and if that replaces every QStandardPaths::foo() call with (say) a KStandardLocations::foo() call we could just as well use an additional link library rather than a plugin.

But ultimately I'm thinking of a plugin that provides alternative methods in QStandardPaths which user code could decide to use (or not) and which would by default just proxy their standard counterparts. Such a system would provide the same kind of flexibility as a solution where qt.conf is used, but in a way that is probably easier to deploy on other platforms, less easy to tinker with by mere mortal endusers and that probably requires touching less files. It might thus be easier to upstream.

A long time ago David Faure pointed out to me that an application-wide solution that toggles QSP behaviour throughout an application is best avoided as it has the potential of doing that toggle at any time when an application is running. Instead, every dynamically loadable component should have its own, fixed QSP behaviour. (This is of course relevant mostly if not only for "linuxy" centralised installs.)

[Dr.-Ing. Christoph Cullmann]

Hi,

Post by Ben Cooksley
On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier

Post by Thomas Friedrichsmeier
Hi,
On Thu, 03 May 2018 16:01:46 +0200

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
plan is actually to get David Faure to champion a solution (whichever
one) into Qt.

I can be wrong, but I don't see that happening beyond his general support.

well, let's hear him on this.

Post by René J.V. Bertin
I don't see what Unix does here. /Library/Application Support is just
the OS X equivalent of $prefix/share and maybe parts of
$prefix/libexec; both are central, shared location that have a
per-user equivalent (~/Library/Application Support and
~/.local/share). The limitations I see are purely and only related to
the way software has to be installed.

Yes, it's about the installation options. It is _primarily_ about the
fact that the one theorically possible installation option on Mac
matches with _none_ of the major available solutions (macports,
homebrew, craft). It is also about the fact that this installation
option causes problems, e.g. if users are to mix MacPorts and a
single-app-bundled KF5 app, or two versions of a KF5 app, or many other
not-so-unlikely scenarios.
Unix does support installing software to any prefix
(because it provides a way to customize lookup). Windows supports
installing software to any prefix (because it includes an
installation-relative path in the list of paths for
GenericDataLocation). Mac does not, because it lacks both (even though
installation-relative paths are available in other QStandardPaths).

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
It's not even hard for individual apps, today. Just compile in all
data files as qresources, and the problem is done with.

That may be true for resources that contain no executable native
code, but it probably won't work for libraries and plugins.

Those can be handled in qt.conf (and macdeployqt auto-generates an
appropriate qt.conf, for instance). That may or may not be our favorite
solution, but it is an already available option (and if there is a
concern that this could cause problems for monolithic (macports-style)
installations, qt.conf can even be provided as a per-app qresource,
which in principle, should be possible to generate, and compile into
each binary, automatically.

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
A fully packaged Kate is available at 88MB download for instance,
despite including a whole bunch of frameworks.

But with that you get a version of the application that just drops
the functionality [...]

I just don't want to get into that discussion, at all, here. I
absolutely see use-cases for both "monolithic" and "single app"
installations. So both should be supported. Currently neither is. So
the issue at hand isn't to discuss which of the two is superior, but to
find out how to fix the problem affecting them both.

It is worth noting that at one point we had a much lighter (only about
20MB or so I think) DMG however that broke icon support.
There is definitely space for optimisation here in terms of what we're
including in the .app bundle.
If we end up going down the QRC route (which also has the advantage of
improving speed as it eliminates the large number of small files) then
we can also ensure the icons are kept compressed on disk.

Post by Thomas Friedrichsmeier

Post by René J.V. Bertin

Post by Thomas Friedrichsmeier
Again, I think this is not an either-or situation. Both centralized
and standalone app installations are affected by this problem, unless
relying on .qrc files, exclusively.

There *is* an either-or situation: work with the existing
QStandardPaths or don't. Kate.app shows that the former is not
impossible, and it has the big advantage that you do everything
in-house without modifying Qt.

Circumventing the problem by not installing files. Since I found out
about this, I sleep a lot better as maintainer of RKWard, but it still

I still think going the QRC way is the simplest solution.
With that, at least in Kate (that uses a lot of frameworks), I am not aware
of any real remaining issue with qstandardpaths.

You can go the monolithic app bundle way with that or the "I share stuff" way
like on Linux.

Actually all frameworks I patched use the QRC way on Linux nowadays, too,
which IMHO has no real drawbacks and even avoids a lot of hassle during
deployment for e.g. Windows (where a lot of small files do suck incredibly
more than on Unices with a decent filesystem).

Much harder is the issue of deploying all plugins/... that are bundled with
the application in a app bundle, as e.g. stuff like my macqtdeploy patch
to patch the kio plugins and co. just rot away in the qt bug tracker.

https://bugreports.qt.io/browse/QTBUG-48836

Greetings
Christoph

[Ben Cooksley]

On Thu, May 3, 2018 at 8:02 PM, Thomas Friedrichsmeier

Post by Thomas Friedrichsmeier
Hi!

Hi all,

Post by Thomas Friedrichsmeier
KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
that location is problematic on Mac because it refers to absolute,
non-customizable system paths, only. We'd like to avoid patching Qt, so
we are looking to come up with a solution that has a chance of getting
upstreamed.
Feel free to skip over any sections that you are already familiar with.
-- Usage of GenericDataLocation in KF5 --
Many KF5 frameworks store data or look for data in
QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
are just two such examples, which, importantly, are designed such that
users (people or applications) can supply additional data files in
specific directories, such as returned by
QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
QStringLiteral("kxmlgui5/"))
Using QStandardPaths::GenericDataLocation has also been suggested
in places such as
https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .
Nowadays compiled in qrc resources are used instead of installed files,
increasingly, and this removes any worries about file system paths.
However, due to the above we should assume that use of
QStandardPaths::GenericDataLocation is wide-spread in KF5-based
applications, and must remain supported.
-- The situation on Linux --
On Unix systems (not Mac), the paths returned for
QStandardPaths::GenericDataLocation correspond to those of the
XDG_DATA_DIRS environment variable. These will usually be system wide
paths including /usr/share and /usr/local/share. Since the typical
application deployment happens to one of those directories, installed
files will be accessible to frameworks out of the box. In case of
non-standard installation paths, modifying XDG_DATA_DIRS provides a
straight-forward mechanism to adjust lookup.
-- The situation on Windows --
On Windows, GenericDataLocation is mapped to the system specific
"FOLDERID_LocalAppData", which includes some system wide paths, but
also "<APPDIR>", and "<APPDIR>/data", where <APPDIR> is the location of
the binary. While these paths are not directly controllable, having the
relative path in them allows
- installing a group of applications sharing the same file or
- installing a single application (with supporting frameworks)
both to arbitrary installation paths. The only thing that is not -
easily - possible compared to Unix, is having two Applications
installed to two separate installation root directories, and still
sharing the same data files.
The expected filesystem layout is somewhat different from Unix (data
instead of share), but those differences can be handled (and are
https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstallDirs.cmake#L529
. ("<APPDIR>/data" is the path chosen from the available options).
-- The situation on MacOS --
On MacOS, GenericDataLocation is mapped to the system specific
"NSApplicationSupportDirectory", which contains the paths
"/Library/Application Support/" and "~/Library/Application Support/".
Notably, any installation-relative path is missing, as are any options
for customization.
As an interesting aside, the handling is different for AppDataLocation
and AppLocalDataLocation, which add the relative path
"<APPDIR>/../Resources/" (and also append "<APPNAME>" to all paths).
This matches with the notion that data either belongs to exactly one
application (AppDataLocation) or to _all_ applications on the system
(GenericDataLocation), but it does not support the notion that there
may be a group/bundle of applications sharing their data, or that a
single application may look up installation-relative data in any other
than "<APPDIR>/../Resources/<APPNAME>". (Importantly not in
"<APPDIR>/../Resources/kxmlgui5", for intance).
We'll talk about this in a bit, but for now, back to the
This setup severely limits the viable deployment options for KF5-based
applications on Mac to basically just one: They must be installed (or
at least their data must be installed) to a fixed central system
folder. Unfortunately, this conflicts with all major existing
- MacPorts: Uses a centralized installation, _but_ inside an own
installation root (/opt/local/ by default), and is designed not to
touch any paths outside that installation root. MacPorts _still_ does
not have any official KF5 packages, and the ones provided by our own
kde-mac people rely on a patched Qt (details in a bit).
- Homebrew: Also a centralized installation, also inside its own
installation root (typically in a user's home directory). Can cope
better because it is not afraid to create symlinks outside its
installation root (we'll discuss that, too).
- Craft ("normal installation"): Again, a centralized installation,
typically inside a user's home directory. No real solution, to the
problem, yet (the reason for this mail).
- Craft ("application bundle images"): Craft can also create .dmg
packages from single applications (and their dependencies), which
provides for very easy drag-and-drop application installers (much
like a .tar.gz or .zip). Right now these support installation anywhere
(system or user folders), without the need to touch anything outside
that single installation directory. A feature that would certainly be
nice to keep. No solution to the GenericDataLocation-problem, yet.
-- Existing approaches to the problem --
--- Symlinks ---
As a solution that does not require any changes to Qt, an installation
routine could place symlinks to the real installation directories
in /Library/Application Support/ or ~/Library/Application Support/ .
1. It requires "smart" installers and uninstallers, will not work with
plain .dmg's.
2. Once installed, an application cannot trivially be moved, any more,
as that would also require adjusting the symlinks.
3. It only really works for a single centralized installation. However,
if installing two KF5-based applications AppA and AppB to separate
installation roots, where would - for instance - ~/Library/Application
Support/kxmlgui5 point to? A directory inside AppA's installation
root, or inside AppB's installation root? It cannot be both.
--- Patching in Unix-like paths ---
The "kde-mac/Macports" people have created a patch to QStandardPaths
that will essentially enable unix-like paths on request of the app. The
https://github.com/RJVB/macstrop/blob/master/aqua/qt5-kde/files/qt593/fix-qstandardpaths6.patch
A stale Qt-Ticket is here: https://bugreports.qt.io/browse/QTBUG-44473 .
I do not want to (and am not qualified to) go into the details of this
rather large patch, but I think for the purpose of this discussion, the
key idea of the approach can be summarized as "include the
XDG_DATA_DIRS in GenericDataLocation paths, as on Unix".
The advantage of this approach is that it provides a mechanism for the
using application to adjust QStandardPaths-behavior to whatever is
1. A rather large patch
2. There may be a concern about side-effects, as the behavior of Qt
will depend on an environment variable where it did not before, with
_potential_ interaction between separate installations.
3. Will also need some "cooperation" from the calling app in order to
work for all installation scenarios (XDG_DATA_DIRS will have to be
set/adjusted, and an extra setup call may be needed to request
"Unix-behavior" from QStandardPaths).
--- Patching in an installation-relative path ---
A much smaller patch
(https://cgit.kde.org/craft-blueprints-kde.git/tree/libs/qt5/qtbase/fix_GenericDataLocation_mac.patch)
could be used to add "<APPDIR>/../Resources" to the end of the path
list returned for GenericDataLocation. This would allow to locate
resources in any installation root (be it centralized or single-app),
albeit in a non-configurable relative path inside the installation
root. This would thus need support from ECM to adjust the
installation-layout, accordingly, _or_ the addition of a symlink
- _inside_ the installation root, this time - e.g. from ROOT/Resources
to ROOT/share .
Not sure about the MacPorts-take on doing something like this, but for
Homebrew and Craft (both setups), this should(TM) be no problem. As the
new search directory is added at the end of the path list, _and_ it is
at an installation-relative path (where only files belonging to that
installation will be found), the potential for unexpected side-effects
should be strictly limited.
As one drawback I can think of, handling supporting binaries, e.g. in
"libexec" may cause a headache, although this, too should probably be
solvable by appropriate adjustments of the installation layout or
symlinks.
-- Other ideas --
Another idea would be to allow qt.conf to influence the paths returned
for GenericDataLocation. As qt.conf can be placed next to the
application to deploy, lookup behavior could be adjusted in a
straight-forward way, limited to each installation.
As, so far, qt.conf does not affect QStandardPaths at all, however,
such a patch would probably be rather intrusive in itself. (Also, so
far, qt.conf only support one path per type, not a list of paths).
-- Other things to consider --
Essentially the same thoughts probably apply to GenericConfigLocation.
I left that out for now, in order to not make the discussion even more
complex. And of course the solution should be transferrable to further
platforms beyond Unix/Windows/Mac, if needed.
-- The bottom line --
Ok, sorry for the long mail. Long story short: We need to come up with
a solution, and it best be a solution that
a) can be upstreamed to Qt
b) works out for MacPorts, Homebrew, _and_ Craft.
So does any of the approaches outlined above qualify for these
conditions, or what else could be a viable route to take?

Just thought i'd comment on the experience we had when we previously
tried to resolve this issue.

Last time around the Qt Core maintainer vetoed our attempts to get
this changed, and forced us to liase with a Qt Company employee whose
responsiveness was lacking (to say the least). Suffice to say, that
effectively killed the effort last time round.

Best wishes with getting this sorted out this time around!

Post by Thomas Friedrichsmeier
Regards
Thomas

Cheers,
Ben

[René J.V. Bertin]

On Thursday May 03 2018 22:22:30 Ben Cooksley wrote:

Hi,

Post by Ben Cooksley
Last time around the Qt Core maintainer vetoed our attempts to get
this changed, and forced us to liase with a Qt Company employee whose
responsiveness was lacking (to say the least). Suffice to say, that
effectively killed the effort last time round.

A Qt Core maintainer who isn't a Qt employee?

IIRC, David Faure is the original author of the QSP class, but maybe not of the actual Mac implementation. David was constructive enough in the past in his feedback on my patch, though we did have some disagreements and we never got around to sitting down to get it sorted out once and for all.

Note that I wouldn't really mind upstreaming only a minimal change to QSP, as long as that doesn't make achieving my patch more complicated (or worse, requires a serious overhaul of all my current KF5 ports).
I really am quite happy with how my current patch works; it allows all KF5 software that doesn't do Mac-incompatible things to work (almost) "out of the box". It also allows me to focus on function over form so to speak ("form" being the way how/where things are installed/bundled).

R.

[Boudewijn Rempt]

Thanks for this analysis. I knew something was wrong, but never dug in deeply
enough to figure out why I was having so much trouble. So I've got a bunch of
workarounds in https://phabricator.kde.org/source/krita/browse/master/libs/
widgetutils/KoResourcePaths.cpp ...

Post by Thomas Friedrichsmeier
Hi!
KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
that location is problematic on Mac because it refers to absolute,
non-customizable system paths, only. We'd like to avoid patching Qt, so
we are looking to come up with a solution that has a chance of getting
upstreamed.
Feel free to skip over any sections that you are already familiar with.
-- Usage of GenericDataLocation in KF5 --
Many KF5 frameworks store data or look for data in
QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
are just two such examples, which, importantly, are designed such that
users (people or applications) can supply additional data files in
specific directories, such as returned by
QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
QStringLiteral("kxmlgui5/"))
Using QStandardPaths::GenericDataLocation has also been suggested
in places such as
https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .
Nowadays compiled in qrc resources are used instead of installed files,
increasingly, and this removes any worries about file system paths.
However, due to the above we should assume that use of
QStandardPaths::GenericDataLocation is wide-spread in KF5-based
applications, and must remain supported.
-- The situation on Linux --
On Unix systems (not Mac), the paths returned for
QStandardPaths::GenericDataLocation correspond to those of the
XDG_DATA_DIRS environment variable. These will usually be system wide
paths including /usr/share and /usr/local/share. Since the typical
application deployment happens to one of those directories, installed
files will be accessible to frameworks out of the box. In case of
non-standard installation paths, modifying XDG_DATA_DIRS provides a
straight-forward mechanism to adjust lookup.
-- The situation on Windows --
On Windows, GenericDataLocation is mapped to the system specific
"FOLDERID_LocalAppData", which includes some system wide paths, but
also "<APPDIR>", and "<APPDIR>/data", where <APPDIR> is the location of
the binary. While these paths are not directly controllable, having the
relative path in them allows
- installing a group of applications sharing the same file or
- installing a single application (with supporting frameworks)
both to arbitrary installation paths. The only thing that is not -
easily - possible compared to Unix, is having two Applications
installed to two separate installation root directories, and still
sharing the same data files.
The expected filesystem layout is somewhat different from Unix (data
instead of share), but those differences can be handled (and are
https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstal
lDirs.cmake#L529 . ("<APPDIR>/data" is the path chosen from the available
options).
-- The situation on MacOS --
On MacOS, GenericDataLocation is mapped to the system specific
"NSApplicationSupportDirectory", which contains the paths
"/Library/Application Support/" and "~/Library/Application Support/".
Notably, any installation-relative path is missing, as are any options
for customization.
As an interesting aside, the handling is different for AppDataLocation
and AppLocalDataLocation, which add the relative path
"<APPDIR>/../Resources/" (and also append "<APPNAME>" to all paths).
This matches with the notion that data either belongs to exactly one
application (AppDataLocation) or to _all_ applications on the system
(GenericDataLocation), but it does not support the notion that there
may be a group/bundle of applications sharing their data, or that a
single application may look up installation-relative data in any other
than "<APPDIR>/../Resources/<APPNAME>". (Importantly not in
"<APPDIR>/../Resources/kxmlgui5", for intance).
We'll talk about this in a bit, but for now, back to the
This setup severely limits the viable deployment options for KF5-based
applications on Mac to basically just one: They must be installed (or
at least their data must be installed) to a fixed central system
folder. Unfortunately, this conflicts with all major existing
- MacPorts: Uses a centralized installation, _but_ inside an own
installation root (/opt/local/ by default), and is designed not to
touch any paths outside that installation root. MacPorts _still_ does
not have any official KF5 packages, and the ones provided by our own
kde-mac people rely on a patched Qt (details in a bit).
- Homebrew: Also a centralized installation, also inside its own
installation root (typically in a user's home directory). Can cope
better because it is not afraid to create symlinks outside its
installation root (we'll discuss that, too).
- Craft ("normal installation"): Again, a centralized installation,
typically inside a user's home directory. No real solution, to the
problem, yet (the reason for this mail).
- Craft ("application bundle images"): Craft can also create .dmg
packages from single applications (and their dependencies), which
provides for very easy drag-and-drop application installers (much
like a .tar.gz or .zip). Right now these support installation anywhere
(system or user folders), without the need to touch anything outside
that single installation directory. A feature that would certainly be
nice to keep. No solution to the GenericDataLocation-problem, yet.
-- Existing approaches to the problem --
--- Symlinks ---
As a solution that does not require any changes to Qt, an installation
routine could place symlinks to the real installation directories
in /Library/Application Support/ or ~/Library/Application Support/ .
1. It requires "smart" installers and uninstallers, will not work with
plain .dmg's.
2. Once installed, an application cannot trivially be moved, any more,
as that would also require adjusting the symlinks.
3. It only really works for a single centralized installation. However,
if installing two KF5-based applications AppA and AppB to separate
installation roots, where would - for instance - ~/Library/Application
Support/kxmlgui5 point to? A directory inside AppA's installation
root, or inside AppB's installation root? It cannot be both.
--- Patching in Unix-like paths ---
The "kde-mac/Macports" people have created a patch to QStandardPaths
that will essentially enable unix-like paths on request of the app. The
https://github.com/RJVB/macstrop/blob/master/aqua/qt5-kde/files/qt593/fix-qs
https://bugreports.qt.io/browse/QTBUG-44473 .
I do not want to (and am not qualified to) go into the details of this
rather large patch, but I think for the purpose of this discussion, the
key idea of the approach can be summarized as "include the
XDG_DATA_DIRS in GenericDataLocation paths, as on Unix".
The advantage of this approach is that it provides a mechanism for the
using application to adjust QStandardPaths-behavior to whatever is
1. A rather large patch
2. There may be a concern about side-effects, as the behavior of Qt
will depend on an environment variable where it did not before, with
_potential_ interaction between separate installations.
3. Will also need some "cooperation" from the calling app in order to
work for all installation scenarios (XDG_DATA_DIRS will have to be
set/adjusted, and an extra setup call may be needed to request
"Unix-behavior" from QStandardPaths).
--- Patching in an installation-relative path ---
A much smaller patch
(https://cgit.kde.org/craft-blueprints-kde.git/tree/libs/qt5/qtbase/fix_Gene
ricDataLocation_mac.patch) could be used to add "<APPDIR>/../Resources" to
the end of the path list returned for GenericDataLocation. This would allow
to locate
resources in any installation root (be it centralized or single-app),
albeit in a non-configurable relative path inside the installation
root. This would thus need support from ECM to adjust the
installation-layout, accordingly, _or_ the addition of a symlink
- _inside_ the installation root, this time - e.g. from ROOT/Resources
to ROOT/share .
Not sure about the MacPorts-take on doing something like this, but for
Homebrew and Craft (both setups), this should(TM) be no problem. As the
new search directory is added at the end of the path list, _and_ it is
at an installation-relative path (where only files belonging to that
installation will be found), the potential for unexpected side-effects
should be strictly limited.
As one drawback I can think of, handling supporting binaries, e.g. in
"libexec" may cause a headache, although this, too should probably be
solvable by appropriate adjustments of the installation layout or
symlinks.
-- Other ideas --
Another idea would be to allow qt.conf to influence the paths returned
for GenericDataLocation. As qt.conf can be placed next to the
application to deploy, lookup behavior could be adjusted in a
straight-forward way, limited to each installation.
As, so far, qt.conf does not affect QStandardPaths at all, however,
such a patch would probably be rather intrusive in itself. (Also, so
far, qt.conf only support one path per type, not a list of paths).
-- Other things to consider --
Essentially the same thoughts probably apply to GenericConfigLocation.
I left that out for now, in order to not make the discussion even more
complex. And of course the solution should be transferrable to further
platforms beyond Unix/Windows/Mac, if needed.
-- The bottom line --
Ok, sorry for the long mail. Long story short: We need to come up with
a solution, and it best be a solution that
a) can be upstreamed to Qt
b) works out for MacPorts, Homebrew, _and_ Craft.
So does any of the approaches outlined above qualify for these
conditions, or what else could be a viable route to take?
Regards
Thomas

[René J.V. Bertin]

One more thought about this whole issue: it could have been avoided completely if KDE4's predecessor of QStandardPaths had been preserved, or if a KStandardPaths class were an acceptable thing to reintroduce.

Such a class probably wouldn't even have to be part of an obligatory central KF5 library. It could be part of KCoreAddons, and I think it would be possible to use it even from code that doesn't do so explicitly with CPPFLAGS=-DQStandardPaths=KStandardPaths .

R.