ossimFileWalker Class Reference

Utility class to walk through directories and get a list of files to process. More...

#include <ossimFileWalker.h>

Classes
class	ossimFileWalkerJob
	Private ossimJob class. More...
class	ossimFileWalkerJobCallback
	Private ossimJobCallback class. More...

Public Member Functions
	ossimFileWalker ()
	default constructor More...
	~ossimFileWalker ()
	destructor More...
void	walk (const std::vector< ossimFilename > &files)
	Takes an array of files. More...
void	walk (const ossimFilename &root)
	This will walk "root" and execute a callback for each file found. More...
void	setFileProcessor (ossimFileProcessorInterface *fpi)
	Sets ossimFileProcessorInterfacecallback method to process files. More...
const std::vector< std::string > &	getFilteredExtensions () const
std::vector< std::string > &	getFilteredExtensions ()
	Non const method to allow access for adding or deleting extensions from the list. More...
void	dumpFilteredExtensionList () const
	Dumps filtered image extenstions to std out. More...
void	initializeDefaultFilterList ()
	Initializes the filter list with a default set of filtered out file names. More...
void	setRecurseFlag (bool flag)
	Sets recurse flag. More...
void	setWaitOnDirFlag (bool flag)
	Sets waitOnDir flag. More...
void	setAbortFlag (bool flag)
	If set to true this stops files walking (aborts). More...
void	setNumberOfThreads (ossim_uint32 nThreads)
	Sets the max number of threads(jobs) to run at one time. More...

Private Member Functions
void	walkDir (const ossimFilename &dir)
	Processes files in directory. More...
bool	isFiltered (const ossimFilename &f) const
	Convenience method for file walker code to check file to see is it should be processed. More...
bool	isDotFile (const ossimFilename &f) const
	isDotFile method. More...

Private Attributes
ossimFileProcessorInterface *	m_fileProcessor
	Callback to method to process a file. More...
std::shared_ptr< ossimJobMultiThreadQueue >	m_jobQueue
std::vector< std::string >	m_filteredExtensions
bool	m_recurseFlag
bool	m_waitOnDirFlag
bool	m_abortFlag
std::mutex	m_mutex

Detailed Description

Utility class to walk through directories and get a list of files to process.

For each file found the ossimFileProcessorInterface::processFile method is excecuted. Internally the processFile calls are placed in a job queue.

Typical usage (snip from ossimTiledElevationDatabase):

ossimFileWalker* fw = new ossimFileWalker(); fw->initializeDefaultFilterList(); fw->setFileProcessor( this ); fw->walk(f);

Definition at line 46 of file ossimFileWalker.h.

Constructor & Destructor Documentation

ossimFileWalker()

ossimFileWalker::ossimFileWalker

(

)

default constructor

Definition at line 28 of file ossimFileWalker.cpp.

   : m_fileProcessor(0),
     m_jobQueue(std::make_shared<ossimJobMultiThreadQueue>(std::make_shared<ossimJobQueue>(), 1)),     
     m_filteredExtensions(0),
     m_recurseFlag(true),
     m_waitOnDirFlag(false),
     m_abortFlag(false),
     m_mutex()
{
}

~ossimFileWalker()

ossimFileWalker::~ossimFileWalker

(

)

destructor

Definition at line 39 of file ossimFileWalker.cpp.

References m_jobQueue.

{
   m_jobQueue = 0; // Not a leak, ref pointer.
}

Member Function Documentation

dumpFilteredExtensionList()

void ossimFileWalker::dumpFilteredExtensionList

(

)

const

Dumps filtered image extenstions to std out.

Definition at line 406 of file ossimFileWalker.cpp.

References m_filteredExtensions, ossimNotify(), and ossimNotifyLevel_NOTICE.

{
   ossimNotify(ossimNotifyLevel_NOTICE) << "Filtered extension list:\n";
   std::vector<std::string>::const_iterator i = m_filteredExtensions.begin();
   while ( i != m_filteredExtensions.end() )
   {
      ossimNotify(ossimNotifyLevel_NOTICE) << (*i) << "\n";
      ++i;
   }
   ossimNotify(ossimNotifyLevel_NOTICE) << std::endl;
}

getFilteredExtensions() [1/2]

const std::vector< std::string > & ossimFileWalker::getFilteredExtensions

(

)

const

Returns

The list of filtered out files.

Definition at line 396 of file ossimFileWalker.cpp.

References m_filteredExtensions.

{
   return m_filteredExtensions;
}

getFilteredExtensions() [2/2]

std::vector< std::string > & ossimFileWalker::getFilteredExtensions

(

)

Non const method to allow access for adding or deleting extensions from the list.

The list is used by the private isFiltered method to avoid trying to process unwanted files.

Definition at line 401 of file ossimFileWalker.cpp.

References m_filteredExtensions.

{
   return m_filteredExtensions;
}

initializeDefaultFilterList()

void ossimFileWalker::initializeDefaultFilterList

(

)

Initializes the filter list with a default set of filtered out file names.

Definition at line 418 of file ossimFileWalker.cpp.

References m_filteredExtensions, and m_mutex.

Referenced by ossimImageElevationDatabase::loadFileMap().

{
   m_mutex.lock();
   
   // Common extensions to filter out, most common first.
   m_filteredExtensions.push_back(std::string("ovr"));
   m_filteredExtensions.push_back(std::string("omd"));
   m_filteredExtensions.push_back(std::string("his"));
   m_filteredExtensions.push_back(std::string("geom"));
   
   // The rest alphabetical.
   m_filteredExtensions.push_back(std::string("aux"));
   m_filteredExtensions.push_back(std::string("bin"));
   m_filteredExtensions.push_back(std::string("dbf"));
   m_filteredExtensions.push_back(std::string("h5"));
   m_filteredExtensions.push_back(std::string("hdr"));
   m_filteredExtensions.push_back(std::string("jgw"));
   m_filteredExtensions.push_back(std::string("jpw"));   
   m_filteredExtensions.push_back(std::string("kwl"));
   m_filteredExtensions.push_back(std::string("log"));
   m_filteredExtensions.push_back(std::string("man"));
   m_filteredExtensions.push_back(std::string("ocg"));
   m_filteredExtensions.push_back(std::string("out"));
   m_filteredExtensions.push_back(std::string("prj"));
   m_filteredExtensions.push_back(std::string("save"));
   m_filteredExtensions.push_back(std::string("sdw"));
   m_filteredExtensions.push_back(std::string("sh"));
   m_filteredExtensions.push_back(std::string("shp"));
   m_filteredExtensions.push_back(std::string("shx"));
   m_filteredExtensions.push_back(std::string("spec"));
   m_filteredExtensions.push_back(std::string("statistics"));
   m_filteredExtensions.push_back(std::string("tar"));
   m_filteredExtensions.push_back(std::string("til"));   
   m_filteredExtensions.push_back(std::string("tfw"));
   m_filteredExtensions.push_back(std::string("tgz"));
   m_filteredExtensions.push_back(std::string("tmp"));
   m_filteredExtensions.push_back(std::string("txt"));

   m_mutex.unlock();
}

isDotFile()

bool ossimFileWalker::isDotFile ( const ossimFilename &  f ) const

private

isDotFile method.

Parameters

f	File/directory to check.

Returns

true if file is a dot file.

Definition at line 354 of file ossimFileWalker.cpp.

References ossimString::find(), ossimString::find_last_of(), and ossimString::size().

Referenced by isFiltered().

{
   bool result = false;
   
   // Get the size in bytes.
   if ( f.size() )
   {
      std::string::size_type firstDotPos = f.find('.');
      if ( firstDotPos == 0 )
      {
         result = true;
      }
      else if ( firstDotPos != std::string::npos ) // Dot in file.
      {
         // Get the position of first dot from the end.
         std::string::size_type lastDotPos = f.find_last_of('.');
         if ( lastDotPos != std::string::npos )
         {
            // Make copy.
            ossimFilename f2 = f;
            
            // Convert an '\'s to '/'s. 
            // f2.convertBackToForwardSlashes();
            
            // Find the first slash from end.
            std::string::size_type lastSlashPos = f2.find_last_of('/');
            
            if (lastSlashPos != std::string::npos) // Found a slash.
            {
               if ( (lastSlashPos+1) == lastDotPos )
               {
                  // dot in front of slash like /home/foo/.xemacs
                  result = true;
               }
            }
         }
      }
   }

   return result;
}

isFiltered()

bool ossimFileWalker::isFiltered ( const ossimFilename &  f ) const

private

Convenience method for file walker code to check file to see is it should be processed.

Parameters

f	File to check.

Returns

true if f is in filter list, false if not.

Definition at line 311 of file ossimFileWalker.cpp.

References ossimString::c_str(), ossimString::downcase(), ossimFilename::ext(), isDotFile(), m_filteredExtensions, and ossimString::size().

{
   bool result = false;
   if ( file.size() )
   {
      if ( isDotFile(file) )
      {
         result = true;
      }
      else if ( file[file.size()-1] == '~' )
      {
         result = true;
      }
      else 
      {
         std::string ext = file.ext().downcase().c_str();
         if ( ext.size() )
         {
            std::vector<std::string>::const_iterator i = m_filteredExtensions.begin();
            while ( i != m_filteredExtensions.end() )
            {
               if ( ext == (*i) )
               {
                  result = true;
                  break;
               }
               ++i;
            }
         }
      }
   }
#if 0 /* Please leave for debug. (drb) */
   if(traceDebug())
   {
      ossimNotify(ossimNotifyLevel_DEBUG)
         << "ossimFileWalker::isFiltered file " << (result?"filtered: ":"not filtered: ")
         << file << "\n";
   }
#endif
   
   return result;
}

setAbortFlag()

void ossimFileWalker::setAbortFlag

(

bool

flag

)

If set to true this stops files walking (aborts).

Parameters

flag	True to abort current "walk".

Definition at line 473 of file ossimFileWalker.cpp.

References m_abortFlag, and m_mutex.

{
   m_mutex.lock();
   m_abortFlag = flag;
   m_mutex.unlock();
}

setFileProcessor()

void ossimFileWalker::setFileProcessor

(

ossimFileProcessorInterface *

fpi

)

Sets ossimFileProcessorInterfacecallback method to process files.

Parameters

fpi	ossimFileProcessorInterface pointer

Definition at line 487 of file ossimFileWalker.cpp.

References m_fileProcessor, and m_mutex.

Referenced by ossimImageElevationDatabase::loadFileMap().

{
   m_mutex.lock();
   m_fileProcessor = fpi;
   m_mutex.unlock();
}

setNumberOfThreads()

void ossimFileWalker::setNumberOfThreads

(

ossim_uint32

nThreads

)

Sets the max number of threads(jobs) to run at one time.

Definition at line 480 of file ossimFileWalker.cpp.

References m_jobQueue, and m_mutex.

{
   m_mutex.lock();
   m_jobQueue->setNumberOfThreads(nThreads);
   m_mutex.unlock();
}

setRecurseFlag()

void ossimFileWalker::setRecurseFlag

(

bool

flag

)

Sets recurse flag.

If set to true this stops recursing of the current directory. Defaulted to true in the constructor. Typically used to indicate the directory being processed holds a directory based image, e.g. RPF data.

Parameters

flag	True to recurse, false to stop recursion of current directory.

Definition at line 459 of file ossimFileWalker.cpp.

References m_mutex, and m_recurseFlag.

{
   m_mutex.lock();
   m_recurseFlag = flag;
   m_mutex.unlock();
}

setWaitOnDirFlag()

void ossimFileWalker::setWaitOnDirFlag

(

bool

flag

)

Sets waitOnDir flag.

If set to true each directory is processed in entirety before sub directories are recursed. This allows callers of setRecurseFlag to disable directory walking. If your code is calling setRecurseFlag this flag should be set to true.

Parameters

flag	true to wait, false to not wait. Defaulted to false in the constructor.

Definition at line 466 of file ossimFileWalker.cpp.

References m_mutex, and m_waitOnDirFlag.

{
   m_mutex.lock();
   m_waitOnDirFlag = flag;
   m_mutex.unlock();
}

walk() [1/2]

void ossimFileWalker::walk

(

const std::vector< ossimFilename > &

files

)

Takes an array of files.

For each file in array: If files is a directory, it will walk it. Files found in walk or files(not directories) in the array will be processed via a job queue.

Files are filter prior to the callback execution. The filtering is to eliminate sending unwanted files to the callback. There is a default filter table. This can be edited by calling the non-const getFilteredExtensions method.

Each callback is placed in a threaded job queue. So users should ensure their callback is thread safe.

Definition at line 44 of file ossimFileWalker.cpp.

Referenced by ossimImageElevationDatabase::loadFileMap().

{
   static const char M[] = "ossimFileWalker::walk";
   if(traceDebug())
   {
      ossimNotify(ossimNotifyLevel_DEBUG) << M << " entered\n";
   }

   if ( files.size() )
   {
      std::vector<ossimFilename>::const_iterator i = files.begin();
      while ( i != files.end() )
      {
         // Must have call back set at this point.
         if ( !m_abortFlag && m_fileProcessor )
         {
            ossimFilename file = (*i).expand();
            if ( file.size() && file.exists() )
            {
               if ( file.isDir() ) // Directory:
               {
                  walkDir(file);
               }  
               else // File:
               {
                  if ( isFiltered(file) == false )
                  {
                     if(traceDebug())
                     {
                        ossimNotify(ossimNotifyLevel_DEBUG)
                           << "Making the job for: " << (*i) << std::endl;
                     }
                     
                     // Make the job:
                     std::shared_ptr<ossimFileWalkerJob> job =
                        std::make_shared<ossimFileWalkerJob>( m_fileProcessor, file );
                     
                     job->setName( ossimString( file.string() ) );
                     
                     job->setCallback( std::make_shared<ossimFileWalkerJobCallback>() );
                     
                     // Set the state to ready:
                     job->ready();
                     
                     // Add job to the queue:
                     m_jobQueue->getJobQueue()->add( job );
                     
                     m_mutex.lock();
                     if ( m_abortFlag )
                     {
                        // Clear out the queue.
                        m_jobQueue->getJobQueue()->clear();
                        
                        break; // Callee set our abort flag so break out of loop.
                     }
                     m_mutex.unlock();
                  }
               }
            }
            else
            {
               ossimNotify(ossimNotifyLevel_WARN)
                  << M << " WARNING: file: \""<< file << "\" does not exist" << std::endl;
            }
         }

         ++i;
      
      } // while ( i != files.end() )

      // FOREVER loop until all jobs are completed.
      while (1)
      {
         ossim::Thread::sleepInMicroSeconds(250);
         if ( m_jobQueue->hasJobsToProcess() == false )
         {
            break;
         }
      }

   } // if ( files.size() )

   if(traceDebug())
   {
      ossimNotify(ossimNotifyLevel_DEBUG) << M << " exiting...\n";
   }  
}

walk() [2/2]

void ossimFileWalker::walk

(

const ossimFilename &

root

)

This will walk "root" and execute a callback for each file found.

Files are filter prior to the callback execution. The filtering is to eliminate sending unwanted files to the callback. There is a default filter table. This can be edited by calling the non-const getFilteredExtensions method.

Each callback is placed in a threaded job queue. So users should ensure their callback is thread safe.

Definition at line 132 of file ossimFileWalker.cpp.

{
   static const char M[] = "ossimFileWalker::walk";
   if(traceDebug())
   {
      ossimNotify(ossimNotifyLevel_DEBUG) << M << " entered root=" << root << "\n";
   }

   // Must have call back set at this point.
   if ( !m_abortFlag && m_fileProcessor )
   {
      ossimFilename rootFile = root.expand();
      if ( rootFile.size() && rootFile.exists() )
      {
         if ( rootFile.isDir() )
         {
            walkDir(rootFile);

            // FOREVER loop until all jobs are completed.
            while (1)
            {
               ossim::Thread::sleepInMicroSeconds(250);
               if ( m_jobQueue->hasJobsToProcess() == false )
               {
                  break;
               }
            }
         }
         else
         {
            // Single file no job queue needed.
            if ( isFiltered( rootFile ) == false )
            {
               m_fileProcessor->processFile( rootFile );
            }
         }
      }
      else
      {
         ossimNotify(ossimNotifyLevel_WARN)
            << M << " WARNING: file: \""<< rootFile << "\" does not exist" << std::endl;
      }
   }
   
   if(traceDebug())
   {
      ossimNotify(ossimNotifyLevel_DEBUG) << M << " exiting...\n";
   }  
}

walkDir()

void ossimFileWalker::walkDir ( const ossimFilename &  dir )

private

Processes files in directory.

If a file in the directory is itself a directory this will do a recursive call to itself. Individual files are processed in a job queue...

Definition at line 182 of file ossimFileWalker.cpp.

{
   static const char M[] = "ossimFileWalker::walkDir";
   if(traceDebug())
   {
      ossimNotify(ossimNotifyLevel_DEBUG)
         << M << " entered...\n" << "processing dir: " << dir << "\n";
   }

   // List of directories in this directory...
   std::vector<ossimFilename> dirs;
   
   // List of files in this directory...
   std::vector<ossimFilename> files;

   m_mutex.lock();
   ossimDirectory d;
   bool ossimDirectoryStatus = d.open(dir);
   m_mutex.unlock();

   if ( ossimDirectoryStatus )
   {
      // Loop to get the list of files and directories in this directory.
      m_mutex.lock();
      ossimFilename f;
      bool valid_file = d.getFirst(f);
      while ( valid_file )
      {
         if ( isFiltered(f) == false )
         {
            if (f.isDir())
            {
               dirs.push_back(f);
            }
            else
            {
               files.push_back(f);
            }
         }
         valid_file = d.getNext(f);
      }
      m_mutex.unlock();

      //---
      // Process files first before recursing directories.  If a file is a directory base image,
      // e.g. RPF, then the callee should call ossimFileWalker::setRecurseFlag to false to
      // stop us from going into sub directories.
      //---
      std::vector<ossimFilename>::const_iterator i = files.begin();
      while (i != files.end())
      {
         if(traceDebug())
         {
            ossimNotify(ossimNotifyLevel_DEBUG) << "Making the job for: " << (*i) << std::endl;
         }
         
         // Make the job:
         std::shared_ptr<ossimFileWalkerJob> job =
            std::make_shared<ossimFileWalkerJob>( m_fileProcessor, (*i) );

         job->setName( ossimString( (*i).string() ) );

         job->setCallback( std::make_shared<ossimFileWalkerJobCallback>() );

         // Set the state to ready:
         job->ready();

         // Add job to the queue:
         m_jobQueue->getJobQueue()->add( job );

         m_mutex.lock();
         if ( m_abortFlag )
         {
            // Clear out the queue.
            m_jobQueue->getJobQueue()->clear();
            
            break; // Callee set our abort flag so break out of loop.
         }
         m_mutex.unlock();

         ++i;
      }

      if ( m_waitOnDirFlag )
      {
         // FOREVER loop until all jobs are completed.
         while (1)
         {
            ossim::Thread::sleepInMicroSeconds(250);
            if ( m_jobQueue->hasJobsToProcess() == false )
            {
               break;
            }
         }
      }

      m_mutex.lock();
      if ( !m_abortFlag && m_recurseFlag )
      {
         // Process sub directories...
         i = dirs.begin();
         while (i != dirs.end())
         {
            m_mutex.unlock();
            walkDir( (*i) );
            m_mutex.lock();

            if ( m_abortFlag )
            {
               break; // Callee set our abort flag so break out of loop.
            }
            ++i;
         }
      }
      m_mutex.unlock();
      
   } // if ( ossimDirectoryOpenStatus )

   // Reset the m_recurseFlag.
   m_mutex.lock();
   m_recurseFlag = true;
   m_mutex.unlock();
   
   if(traceDebug())
   {
      ossimNotify(ossimNotifyLevel_DEBUG) << M << " exited...\n";
   }
}

Member Data Documentation

m_abortFlag

bool ossimFileWalker::m_abortFlag

private

Definition at line 221 of file ossimFileWalker.h.

Referenced by setAbortFlag().

m_fileProcessor

ossimFileProcessorInterface* ossimFileWalker::m_fileProcessor

private

Callback to method to process a file.

Parameters

const

ossimFilename& First parameter(argument) file to process.

Definition at line 216 of file ossimFileWalker.h.

Referenced by ossimFileWalker::ossimFileWalkerJob::run(), and setFileProcessor().

m_filteredExtensions

std::vector<std::string> ossimFileWalker::m_filteredExtensions

private

Definition at line 218 of file ossimFileWalker.h.

Referenced by dumpFilteredExtensionList(), getFilteredExtensions(), initializeDefaultFilterList(), and isFiltered().

m_jobQueue

std::shared_ptr<ossimJobMultiThreadQueue> ossimFileWalker::m_jobQueue

private

Definition at line 217 of file ossimFileWalker.h.

Referenced by setNumberOfThreads(), and ~ossimFileWalker().

m_mutex

std::mutex ossimFileWalker::m_mutex

private

Definition at line 222 of file ossimFileWalker.h.

Referenced by initializeDefaultFilterList(), setAbortFlag(), setFileProcessor(), setNumberOfThreads(), setRecurseFlag(), and setWaitOnDirFlag().

m_recurseFlag

bool ossimFileWalker::m_recurseFlag

private

Definition at line 219 of file ossimFileWalker.h.

Referenced by setRecurseFlag().

m_waitOnDirFlag

bool ossimFileWalker::m_waitOnDirFlag

private

Definition at line 220 of file ossimFileWalker.h.

Referenced by setWaitOnDirFlag().

---

The documentation for this class was generated from the following files:

• ossimFileWalker.h
• ossimFileWalker.cpp