Announcement

#1 2024-01-30 06:27:36

rw22mhhs
Member
2024-01-30
6

Read tags from xmp sidecars

Dear all,

Short story (or question): How can I get the https://github.com/alex-phillips/Piwigo … car-plugin to work?

Long story:

I recently managed to install Piwigo on my Qnap NAS. Really a great tool, even though I would have loved to know some things beforehand, as I got stuck for several days (use "127.0.0.1” instead of “localhost” for the MariaDB connection, do always create a subfolder in “./galleries”, as test images placed directly into the “galleries” folder will not be detected). Anyhow, it is up and running now, I can upload photos, synchronize the galleries folder on my NAS etc.

However, there is one thing: I need to import xmp sidecar metadata. The plan is to use digiKam as management hub. I want to assign keywords in digiKam, stored in xmp sidecar files in order to use them as basis for the assignment of pictures to virtual albums and user profiles. Then synchronize the photo database on the NAS (managed by digiKam and symlinked to ./galleries/) with Piwigo's synchronization function and assign the pictures with certain keywords to profiles (probably using the SmartAlbums plugin, but I’m not there yet).

Why separate sidecar files? I have about 130.000 photos, many of them raw files with more than 100 megapixels, where even the jpeg equivalents are easily above 50 mb. I have more than 170 different profiles with tens of thousands of photos, contained in my more than 20 years old, self-developed php script-based database, which I plan to migrate into xmp sidecars for digiKam/Piwigo. Writing these amounts of metadata into the image files itself is simply not an option, as assigning just one additional profile to the images easily results in having to write dozens of gigabytes of data. Additionally, I figured that digiKam has issues in writing the metadata into certain raw formats.

So I tried Piwigo's read metadata plugin. It did not read the metadata from the xmp sidecars and I understood that its only function is to display such metadata separately and next to the image, but not to e.g. integrate the keywords into Piwigo’s keyword database. Additionally, it does not read xmp sidecar files but only metadata from the photos itself. Is this understanding correct?

Then, I discovered the https://github.com/alex-phillips/Piwigo … car-plugin which can read separate xmp sidecar files. Unfortunately, I cannot get this plugin to work. Even though it is triggered correctly by the synchronization function and reads the metadata correctly from the sidecar files (I performed logging tests which show that it reads all keywords correctly and writes them into a logfile), it does not write them into the Piwigo database. So pictures imported do not show any of the metadata that the plugin was able to write into the logfile. I just took the plugin from GitHub and uploaded it into the plugin directory. Uncommenting the code in the maintain.inc.php did not change anything.

Is there any chance to get this fixed and get the xmp sidecar plugin to work? Or is there any other solution to read metadata from XMP sidecars? I browsed through all other Piwigo plugins I could found...

I am using Piwigo 14.1.0 on a QNAP NAS, PHP 7.4.30, MariaDB 10.5.8, ImageMagick 6.9.4-8.

It would be great if someone can take a look at the xmp sidecar plugin or let me know if there is any other solution for this task.

Many thanks in advance!

Last edited by rw22mhhs (2024-01-30 06:29:21)

Offline

 

#2 2024-01-30 07:31:29

ddtddt
Piwigo Team
Quetigny - France
2007-07-27
7208

Re: Read tags from xmp sidecars

Hi :-)

Can you give me link photo and  xmp sidecars for test


You love Piwigo so don't hesitate to participate, learn more on the "Contribute to Piwigo" page. If you don't have much time for contribution, you can also help the project with a donation.

Offline

 

#3 2024-01-30 08:58:05

rw22mhhs
Member
2024-01-30
6

Re: Read tags from xmp sidecars

Hi,

sure, see https://file.io/lSru98fgJJkq for two sample files.

The amended plugin logged the following:

import_xmp_sidecar called for file: ././galleries/test/DSC_0835.JPG
xmp_keywords: Family
xmp_hierarchical_keywords: _Album|Family

For logging, I added the following to the plugin:

$logFile = "/[...]/logs/xmp-plugin-logfile.log";
    error_log("import_xmp_sidecar called for file: $filename\n", 3, $logFile);
   
...

// Log the extracted data
    foreach ($xmpData as $key => $val) {
        error_log("$key: $val\n", 3, $logFile);
    }
   
The photo is correctly added to the gallery. However, no metadata as listed above can be found within the properties in Piwigo.

Thank you!

Last edited by rw22mhhs (2024-01-30 09:08:23)

Offline

 

#4 2024-02-07 04:49:46

rw22mhhs
Member
2024-01-30
6

Re: Read tags from xmp sidecars

Hi there,

Does anyone have an idea how to achieve the above? Even though I managed to import exif tags that are written by digiKam into the image file, I did not find a way to filter or sort by those tags. They are only shown by the Read Metadata plugin.

I thought that I could import xmp keywords/tags and that they would be shown as tags in Piwigo. Am I too naive?

So, in brief, I would have to get the import_xmp_sidecar plugin to work and then to import the xmp sidecar file's tags as tags into Piwigo. The SmartAlbums plugin seems to provide the functionality I need as the following step. However, the first two steps are missing...

After days and weeks of trying to get this to work, any help is highly appreciated.

Many thanks!

Offline

 

#5 2024-02-10 16:36:08

rw22mhhs
Member
2024-01-30
6

Re: Read tags from xmp sidecars

Okay, I found the solution myself.

I had to insert the following lines into the config.inc.php:

$conf['use_exif'] = true;

$conf['use_exif_mapping'] = array(
  'date_creation' => 'DateTimeOriginal',
  'keywords' => 'xmp_keywords',
  );

Offline

 

#6 Yesterday 06:16:54

ashetl
Member
2024-05-12
2

Re: Read tags from xmp sidecars

Hi rw22mhhs

I'm interested to see that you got this going.

I've been trying the same, for similar reasons as you.

When I attempt to synchronize metadata, I get the following error message (note that references to the XMPTags folder are to where my piwigo-xmp-sidecar-plugin is saved):

Fatal error: Uncaught TypeError: array_merge(): Argument #1 must be of type array, null given in /var/www/piwigo/piwigo/plugins/XMPTags/main.inc.php:80 Stack trace: #0 /var/www/piwigo/piwigo/plugins/XMPTags/main.inc.php(80): array_merge() #1 /var/www/piwigo/piwigo/include/functions_plugins.inc.php(214): import_xmp_sidecar() #2 /var/www/piwigo/piwigo/include/functions_metadata.inc.php(138): trigger_change() #3 /var/www/piwigo/piwigo/admin/include/functions_metadata.php(77): get_exif_data() #4 /var/www/piwigo/piwigo/admin/include/functions_metadata.php(226): get_sync_exif_data() #5 /var/www/piwigo/piwigo/admin/site_reader_local.php(155): get_sync_metadata() #6 /var/www/piwigo/piwigo/admin/site_update.php(862): LocalSiteReader->get_element_metadata() #7 /var/www/piwigo/piwigo/admin.php(346): include('...') #8 {main} thrown in /var/www/piwigo/piwigo/plugins/XMPTags/main.inc.php on line 80

I've been assuming that the error results from how my DigiKam instance has structured the data in its XMP sidecars. I use only a few tags in DigiKam, which are structured hierarchically under auto/, Location/ and People/. Some of the People/ tags include DigiKam face tags and some do not.

Did you find that you needed to alter the script in main.inc.php to align with the DigiKam tagging structure, or otherwise encounter an issue similar to this in getting the plugin to work?

Offline

 

#7 Yesterday 11:44:02

rw22mhhs
Member
2024-01-30
6

Re: Read tags from xmp sidecars

Hi there,

Yes, indeed I inserted some more changes as I ran across similar errors subsequently. For the sake of completeness, here is my complete main.inc.php of the plugin. Please note that it is taylored for nested tags starting with an underscore, such as "_Profile|Jon Doe" or "_Album|Vacation2024":

Code:

<?php

if (!defined('PHPWG_ROOT_PATH')) die('Hacking attempt!');

// Define the path to our plugin.
define('XMPSIDECAR_DIR', basename(dirname(__FILE__)));
define('XMPSIDECAR_PATH', PHPWG_PLUGINS_PATH . XMPSIDECAR_DIR . '/');

add_event_handler(
    'format_exif_data',
    'import_xmp_sidecar'
);

/**
 * Process XMP sidecar files and merge the data into the EXIF data.
 *
 * Function logic was pulled from https://surniaulula.com/2013/apps/wordpress/read-adobe-xmp-xml-in-php/
 * and took some inspiration from http://piwigo.org/forum/viewtopic.php?id=25709
 * amended by rw22mhhs
 * 
 * @param array $exif
 * @param string $filename
 * @return void
 */

function import_xmp_sidecar($exif, $filename)
{
    /* 
    $logFile = "/share/CE_CACHEDEV1_DATA/Web/media/logs/xmp-plugin-logfile.log";
    error_log("#########################################\n", 3, $logFile);
    error_log("import_xmp_sidecar called for file: $filename\n", 3, $logFile);
    */ 
    
    $sidecarFilename = "$filename.xmp";
    if (!file_exists($sidecarFilename)) {
        return $exif;
    }

    $xmpRaw = file_get_contents($sidecarFilename);
    $xmpData = array();

    $tags = [
        'Creator Email'         => '<Iptc4xmpCore:CreatorContactInfo[^>]+?CiEmailWork="([^"]*)"',
        'Owner Name'            => '<rdf:Description[^>]+?aux:OwnerName="([^"]*)"',
        'Creation Date'         => '<rdf:Description[^>]+?xmp:CreateDate="([^"]*)"',
        'Modification Date'     => '<rdf:Description[^>]+?xmp:ModifyDate="([^"]*)"',
        'Label'                 => '<rdf:Description[^>]+?xmp:Label="([^"]*)"',
        'Credit'                => '<rdf:Description[^>]+?photoshop:Credit="([^"]*)"',
        'Source'                => '<rdf:Description[^>]+?photoshop:Source="([^"]*)"',
        'Headline'              => '<rdf:Description[^>]+?photoshop:Headline="([^"]*)"',
        'City'                  => '<rdf:Description[^>]+?photoshop:City="([^"]*)"',
        'State'                 => '<rdf:Description[^>]+?photoshop:State="([^"]*)"',
        'Country'               => '<rdf:Description[^>]+?photoshop:Country="([^"]*)"',
        'Country Code'          => '<rdf:Description[^>]+?Iptc4xmpCore:CountryCode="([^"]*)"',
        'Location'              => '<rdf:Description[^>]+?Iptc4xmpCore:Location="([^"]*)"',
        'Title'                 => '<dc:title>\s*<rdf:Alt>\s*(.*?)\s*<\/rdf:Alt>\s*<\/dc:title>',
        'Description'           => '<dc:description>\s*<rdf:Alt>\s*(.*?)\s*<\/rdf:Alt>\s*<\/dc:description>',
        'Creator'               => '<dc:creator>\s*<rdf:Seq>\s*(.*?)\s*<\/rdf:Seq>\s*<\/dc:creator>',
        'Keywords'              => '<dc:subject>\s*<rdf:Bag>\s*(.*?)\s*<\/rdf:Bag>\s*<\/dc:subject>',
        'Hierarchical Keywords' => '<lr:hierarchicalSubject>\s*<rdf:Bag>\s*(.*?)\s*<\/rdf:Bag>\s*<\/lr:hierarchicalSubject>'
    ];

    foreach ($tags as $key => $regex) {
        $key = "xmp_" . str_replace(' ', '_', strtolower($key));
        $xmpData[$key] = preg_match("/$regex/is", $xmpRaw, $match) ? $match[1] : '';
        if ($key === 'xmp_hierarchical_keywords') {
            // Hierarchical keywords special processing
            $hierarchicalKeywords = preg_match_all("/<rdf:li[^>]*>([^>]*)<\/rdf:li>/is", $xmpData[$key], $match) ? $match[1] : [];
            $filteredKeywords = array_filter($hierarchicalKeywords, function($val) {
                return strpos($val, '_') === 0;
            });
            $xmpData[$key] = implode(',', $filteredKeywords);
        } else {
            // Generic handling for other fields
            $xmpData[$key] = preg_match_all("/<rdf:li[^>]*>([^>]*)<\/rdf:li>/is", $xmpData[$key], $match) ? implode(',', $match[1]) : $xmpData[$key];
        }
    }

    // Remove empty values
    $xmpData = array_filter($xmpData, function($value) {
        return !empty($value);
    });

    $exif = array_merge($exif, $xmpData);
    
    /*
    // Log the extracted data
    foreach ($xmpData as $key => $val) {
        error_log("$key: $val\n", 3, $logFile);
    }

    error_log("----------------------------------------\n", 3, $logFile);
    error_log("Exif data after merging with XMP sidecar data:\n", 3, $logFile);
    foreach ($exif as $key => $val) {
        if (is_array($val)) {
            $val = implode(', ', $val);
        }
        error_log("$key: $val\n", 3, $logFile);
    }
    */
    
    return $exif;
}

The above code adds tags and the description from XMP sidecars to the piwigo database.

And I added this to the /include/functions_metadata.inc.php:

Code:

  if (!$conf['allow_html_in_metadata'])
  {
// inserted rw22mhhs 
   if (!function_exists('strip_html_in_metadata')) { // inserted rw22mhhs 18.04.2024
    function strip_html_in_metadata() {
        function strip_html_in_metadata(&$v, $k)
    {
      $v = strip_tags($v);
    }
   }} // inserted rw22mhhs 
    foreach ($result as $key => $value)
    {
      // in case the origin of the photo is unsecure (user upload), we remove
      // HTML tags to avoid XSS (malicious execution of javascript)
      if (is_array($value))
      {
        array_walk_recursive($value, 'strip_html_in_metadata');
      }
      else
      {
        $result[$key] = strip_tags($value);
      }
    }
  }

  return $result;
}

In /include/functions_tag.inc.php, I changed the following function:

Code:

// inserted by rw22mhhs to filter underscore_tags from user accounts.
function get_available_tags($tag_ids=array())
{
   global $user;
  
  // we can find top fatter tags among reachable images
  $query = '
SELECT tag_id, COUNT(DISTINCT(it.image_id)) AS counter
  FROM '.IMAGE_CATEGORY_TABLE.' ic
    INNER JOIN '.IMAGE_TAG_TABLE.' it
    ON ic.image_id=it.image_id
  WHERE 1=1
  '.get_sql_condition_FandF(
    array(
      'forbidden_categories' => 'category_id',
      'visible_categories' => 'category_id',
      'visible_images' => 'ic.image_id'
      ),
    ' AND '
    );

  if (is_array($tag_ids) and count($tag_ids) > 0)
  {
    $query .= '
    AND tag_id IN ('.implode(',', $tag_ids).')
';
  }

  $query .= '
  GROUP BY tag_id
;';
  $tag_counters = query2array($query, 'tag_id', 'counter');

  if (empty($tag_counters))
  {
    return array();
  }

  $query = '
SELECT *
  FROM '.TAGS_TABLE;
  $result = pwg_query($query);

  $tags = array();
  while ($row = pwg_db_fetch_assoc($result))
  {
    $counter = intval(@$tag_counters[ $row['id'] ]);
    if ($counter) {
        // Inserted by rw22mhhs - exclude Tags with underscore, except for webmasters
        // Ensure that current user is neither administrator nor webmaster
        if (in_array($user['status'], ['admin', 'webmaster']) || strpos($row['name'], '_') !== 0) {
            $row['counter'] = $counter;
            $row['name'] = trigger_change('render_tag_name', $row['name'], $row);
            $tags[] = $row;
        }
    }
  }
  return $tags;  
}

In the local/config/config.inc.php, I added the following lines:

Code:

// use_exif: Use EXIF data during database synchronization with files
// metadata
$conf['use_exif'] = true;

// use_exif_mapping: same behaviour as use_iptc_mapping
// inserted by rw22mhhs
$conf['use_exif_mapping'] = array(
  'date_creation' => 'DateTimeOriginal',
  // 'keywords' => 'xmp_keywords'  
  'keywords' => 'xmp_hierarchical_keywords',
  'comment' => 'xmp_description'  
  );

And BTW, if you want to enable xmp sidecar metadata extraction also for video files, you will have to change this in the /plugins/piwigo-videojs/main.inc.php (currently only implemented for mp4 files):

Code:

function vjs_format_exif_data($exif, $filename, $map)
{
  global $conf, $picture, $prefixeTable;

  // Begin insert rw22mhhs
  // #####################
  
  // $logFile = "/share/CE_CACHEDEV1_DATA/Web/media/logs/xmp-plugin-logfile.log";
  
  // Remove directory part 'pwg_representative'
  $pathWithoutRep = str_replace('/pwg_representative', '', $filename);

  // Check, if path ends with '.jpg' endet, replace by '.mp4'
  if (substr($pathWithoutRep, -4) === '.jpg') {
    $newPath = substr_replace($pathWithoutRep, '.mp4', -4);
  }
  
  // error_log("#########################################\n", 3, $logFile);
    // error_log("vjs-Dateiname: $newPath\n", 3, $logFile);
    
  // Insertd rw22mmhs: Import metadata from the XMP sidecar file if applicable
  $exif = import_xmp_sidecar($exif, $newPath);

  // End insert rw22mhhs
  // ###################

I hope, this helps.

Ah, finally – as I use the tags for assigning images and videos to the profiles, you may want to hide the tags from the user’s GUI. The above changes hide all underscore tags from the user account. However, they tags are still shown in various places In order to change this, use the Advanced Menu Manager to hide the menus on tags. Additionally, I wrote the following plugin to hide the tags on the tags page:

Code:

<?php
defined('PHPWG_ROOT_PATH') or die('Hacking attempt!');

/*
Plugin Name: Hide Tags for Users
Version: 1.0
Description: Hides Tags for users logged, if not loged in as admin or webmaster. Needs to be used together with Advanced Menu Manager in order to additionally hide the Tags menu.
Author: rw22mhhs
*/


add_event_handler('loc_begin_index', 'hide_tags_globally_for_non_admins');
add_event_handler('loc_end_tags', 'hide_tags_for_non_admins');
add_event_handler('loc_end_section_init', 'check_user_status_for_tags');

function check_user_status_for_tags()
{
    global $user, $page, $conf;
    
    // Ensure that current user is neither administrator nor webmaster
    if (!in_array($user['status'], ['admin', 'webmaster'])) {
        // Deactivate tags for search and menu 
        $conf['menubar_tag_cloud_items_number'] = 0;
        $conf['full_tag_cloud_items_number'] = 0;
        $conf['content_tag_cloud_items_number'] = 0;

        // Prevent tags from appearing in search results
        if ($page['section'] == 'tags' || $page['section'] == 'search') {
            $page['items'] = array();
        }
    }
}

function hide_tags_for_non_admins()
{
    global $user, $template;

    if (!in_array($user['status'], ['admin', 'webmaster'])) {
        $template->assign('tags', array());
        $template->assign('letters', array());
    }
}

function hide_tags_globally_for_non_admins()
{
    global $user, $conf;
    
    if (!in_array($user['status'], ['admin', 'webmaster'])) {
        $conf['menubar_tags_enabled'] = false;
    }
}
?>

I hope, this helps.

Best,
rw22mhhs


[erAck: edited to format code in code tags; not only looks better but more important is less error-prone when copy-pasting; and thanks for contributing]

Last edited by erAck (Yesterday 21:36:21)

Offline

 

#8 Yesterday 13:20:22

ashetl
Member
2024-05-12
2

Re: Read tags from xmp sidecars

That's brilliant. Thank you - it'll take me some time to digest but good to have the sense that I'm on the right track and hopefully I'll be able to devise my own solution.

Offline

 

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact