Tom’s recent post about the Perforce Directory Standard (PDS) generated quite a bit of interest. Creating and maintaining a solid branch and directory model is a common SCM challenge, and the PDS is a great starting point.

Another challenge is communicating the PDS to the users. The value of the PDS is greatly enhanced if users understand how it impacts the development process. If you understand the branching patterns in the PDS, you’ll understand why we merge a bug fix through main instead of grabbing it directly from a release branch.

In the interest of using a picture to save a thousand words, I wrote a quick template for a PDS P4V applet. I assumed I was managing three products, using different branching patterns for each product. The applet has a bit of summary information about branching patterns and models, but most importantly, it has pictures! The applet presents simple branch diagrams for the three products, illustrating the flow of change for each product. A couple of screenshots are shown below.

PDS Branch Diagram for Customization Model

The applet and more screenshots are available in the Public Depot. You can customize it for your own branching patterns and products. The diagrams are generated using native HTML canvas capabilities, so you can easily create updated diagrams as your products evolve.

The applet is a very simple example of visualizing the PDS, but it has the advantage of being present in P4V, which is one place developers tend to spend a lot of time.

PS

I’ve updated the Integration Preview applet I wrote about earlier, adding a couple new features. There’s also a simple applet that displays links to the latest Perforce documentation.

Not too long ago, “distributed” in the context of version control referred to location of the users. Lately the term is also being used to define location of the repository, and you have probably seen the advent of distributed version control systems (DVCS) like Git and Mercurial. You are probably quite familiar with Perforce’s centralized architecture: a central repository that all users connect to, and pull the code down to their own respective workspaces. If you are part of a “remote” group of users then you could setup Perforce proxy as an intermediary to mitigate WAN transmission overheads.  In fact most commercial version control products have similar model of a central repository or multiple nodes on the network that users work off of. DVCS products use a different architecture: a repository for each user.

Wait a second – a repository for every user – isn’t that an overkill? Well if you work in an enterprise where controlled development environment is paramount, then probably that’s what you would think. However there are several motivations behind such architecture, mostly driven by the needs of open source software (OSS) development world:

• Better performance: Open source SCM tools generally lack in performance, particularly across WAN, for operations such as commits, viewing revision history, or reverting changes. Having the entire repository local to the user gets around that problem.
• No central control: Unlike commercial software development, OSS community is loosely managed in terms of development standards, procedures and policies.  DVCS does not mandate a canonical development “trunk” branch – it needs to be declared and managed by someone (the curator).
• Working disconnected: OSS community is a loosely-coupled network of individual developers who like to work pretty much on their own, and like the idea of being isolated from others, especially when trying out experimental code branches.

So how does Perforce stack up, you wonder? Overall performance is one of the notable strengths of Perforce, while Perforce Proxy architecture works quite well to support distributed development.  Having a central “gatekeeper” is an organizational choice and Perforce is quite adaptable at instituting as much or as little centralized control over things like user check-ins. As far as working disconnected is concerned, Perforce has recently added newer features to P4V that allow all user actions (adds, edits, deletes) to be cached in a local “offline” repository, to be subsequently merged with the main repository. Besides, in today’s uber-connected world, finding ourselves out of connectivity is an exception than a norm.

DVCS is still an evolving field; tools, procedures and practices in its ecosystem haven’t quite settled down yet. It seems to have broader appeal in the OSS community rather than the commercial enterprise. Not to say that Perforce is not suitable for OSS development; in fact we are happy to provide free Perforce licenses if you develop software that is licensed or otherwise distributed exclusively under an Open Source license.

Warning Readers:  This is a shameless plug for my upcoming webinar, Subversion to Perforce in 30 Minutes.

Perforce is becoming the enterprise corporate standard tool of choice in more organizations, for many good reasons.  The global Perforce Consulting teams help customers migrate to Perforce from a great variety of version control systems as we help drive standardization efforts.  People realize that, while Subversion is a trusty screwdriver, it’s not the sort of power tool you should provide to highly paid programmers to maximize productivity.

Want to be Agile and maximize the amount of work not done?  Use Perforce and reduce merge rework!  Use a tool that totally groks the dark arts of base selection and merge credits — resulting in more accurate first-time merges, saving human effort.

So, instead of being submersed in low-tech tools like the Turtle, learn about migrating to Perforce.  Got gigabytes of files and years of version history in Subversion you don’t know what to do with?  Attend the webinar to get a better sense of your options.

Hope to see you there!

Looking for a little extra Perforce help? Want to make sure your server is backed up and running as efficiently as possible? Perforce Remote Administration is here to help.

The program gives you access to a team of expert Perforce staff members who can act as your Perforce administrators. They access your Perforce setup via secure remote access technologies. By enrolling in the program, you’ll receive several benefits, including daily server monitoring, monthly server reports, and on-demand access to expert assistance.

If you’re like many of our customers, you don’t need to pay much attention to your Perforce server. It just works, so you don’t often need a lot of in-house expertise. But, a few times a year, you may find yourself reaching a bit past your comfort level. Maybe you need to refine your branching model to incorporate a new development effort, set up an automated build system, or you just want to make sure your Perforce server is set up to handle future growth.

If you enroll in P4RA, you can have confidence that we’re keeping an eye on your server, and you can take advantage of our years of Perforce expertise. Not sure how to change your development model to accommodate Agile development, with its rapid sprints? We can help. Need additional expertise in setting up your Perforce server to handle disaster recovery, or have a certain minimum uptime? We can help with that too.

If you’d like to explore this program further for your own environment, write to the Perforce Remote Administration team.

While researchers try to solve the problem of generic, secure remote computability and storage, we Perforce users can take advantage of the advanced features of Tahoe-LAFS to hold the versioned file store of the Perforce server in an otherwise untrusted cloud of remote storage servers. This ability was recently made much simpler via the +X filetype.

What Is Cloud Computing

There has been a lot of buzzword bandying lately of the term “cloud computing.” Embryonic shifts towards remote computing services include the various commercial “cloud” offerings available such as Amazon’s Elastic Compute Cloud, the private cloud Open Source Eucalyptus, and then at a much lower level, parallel computing platforms such as the Open Source Hadoop umbrella. Beyond that are wonderfully advanced concepts that haven’t escaped academic research papers into practical form yet.

As much as these forays may promise to converge into ubiquity, this hasn’t yet happened—but it is threatening to. One of the biggest obstacles of this convergence into a trustworthy general-purpose platform is well-vetted user-controlled strong cryptography wrapping not only sensitive data, but computation involving that data. Promising advances in both of these areas include Tahoe LAFS and Hadoop-on-Tahoe secure remote computation.

Introduction To Tahoe-LAFS

Tahoe-LAFS allows users to store data remotely without having to worry so much about trusting the network, storage nodes, or even machine reliability. Tahoe is one of the most conceptually advanced software storage mechanisms available, and uses cryptographically-sound “capabilities” to facilitate varying levels of trust and permissions. Someone with a “read-only” key might only be able to read an object’s contents. Someone else with a “verify” key might be able to double-check the data is valid. A third person might have “read/write” keys which enable full access to stored objects.

Tahoe-LAFS encrypts a file client-side to produce ciphertext, then splits the ciphertext into generic erasure-coded redundant shares, and then spreads these shares out to remote untrusted nodes to store. Due to the nature of erasure-coding, only x of y shares are necessary to reconstruct the original, so, y-x shares can disappear from a Tahoe grid for any reason and your original file will still be functionally available. The overhead for this is surprisingly light, and x and y are 100% under the user’s control.

Read More

Genesis

It was back in early 2007 that I first had the idea of using HTML in P4V. I was coding the Home Page of the (then new) Administration Tool, and thought “Why am I coding an HTML-like interface in C++/Qt, why not just use HTML?” In this post, I describe the long road that lead to the 10.1 release of the Perforce JavaScript API for the Visual Tools.

Although it was clear to me that HTML was a natural choice for user interface development, there were numerous problems to actually creating a usable implementation- not the least of which was lack of an HTML rendering engine in C++!  I looked into porting the rendering engine from an open source browser, but decided that was too much work to be practical.

Fast forward to the spring of 2008, when Qt, a C++ cross-platform development library we use, released a WebKit integration in version 4.4.

(WebKit is the HTML rendering engine that powers Google Chrome and Apple Safari web browsers.)

Now the dream of using HTML to create interfaces in P4V became a reality, and I checked a tiny prototype into my dev branch. A small group of HTML visionaries formed: Jason Gibson, Bill Baffy and Michael Bishop.

Three Problems

Although we created a working HTML prototype of the Administration Tool’s Home Page running inside P4V, three problems became clear: first, without access to data in the Perforce server, the interface could not be very useful; second, where could the HTML files be stored?; and third, how do we know which HTML file goes where in P4V?

Read More

Recently someone asked me if there is any way to automatically save the merge output files with conflict markers while running p4 resolve. In this scenario, we’re running a big integration, potentially merging hundreds of files at once. We rely on the automatic merge action (p4 resolve -am) where possible, but some of the files have conflicts. When we run into a merge conflict, we need to grab the person who knows that file and can determine how to manually address the conflict. Rather than having that person walk over to our desk, we’d like to send her the file with conflict markers. She can manually edit the file and send it back to us, and we’ll use the edited file to finish the resolve.

After thinking for a few minutes, I remembered that the P4MERGE environment variable lets us solve this problem. P4MERGE, when set, provides an alternative three way merge program for use by p4 resolve when the user selects the m resolve option. We can set that variable to point to a custom script, not just a regular diff/merge utility.

It’s certainly easy to use P4MERGE and shell scripts to accomplish this goal, but the P4Python API lets us develop a more elegant solution. Specifically, we’re going to use the P4.Resolver class. Using P4.Resolver in the P4Python API is even more powerful than setting the P4MERGE environment variable to a custom script. By providing a custom implementation of P4.Resolver, we can use custom logic for handling resolves. We could choose to accept the merged output file for text files, for instance, while choosing the accept theirs action for binary files. In this case, we’ll provide a custom P4.Resolver implementation that archives the merged output file, then skips the resolve.

The example script is available in the Public Depot. The script tries to resolve any files in the workspace that still need resolving. The MyResolver implementation of P4.Resolver simply saves off the merged result file to a temporary directory, after giving it a better name. It then returns s, indicating that the resolve should be skipped. Since run_resolve calls the resolver once for each file, it gives us a one-stop way to archive any files that need resolving.

The script should be run from a client workspace that contains files that need to be resolved. (The script is run standalone, not invoked by setting P4MERGE and running the p4 resolve command.)

Here are the interesting snippets of the script:

# To-do: provide a better archive directory than /tmp
class MyResolver(P4.Resolver):
    def resolve(self, mergeData):
        dst = "/tmp/" + basename(mergeData.your_name)
        copy(mergeData.result_path, dst)
        return "s" 

p4 = P4.P4()
try:
   p4.connect()
   p4.run_resolve ( resolver=MyResolver() );
   p4.disconnect()

We could archive all four files involved in the merge, by enhancing the MyResolver class. It has access to all the elements in the P4.MergeData class.

Mechanically, then, here’s what we’ll do:

The new Perforce server release 2010.1 has just been released; you might have seen the announcements on our website. There have been many internal improvements, most noteworthy for me has been the addition of ‘move -f’, which Sam Stafford wrote about in a recent post here.

There is also a new feature many Perforce users have been asking for for a long time: the ability to search through the content of files on the server. This new feature is called ‘p4 grep’, named after the well-known tool grep of Unix lore.

‘p4 grep’ allows users to use simple file searches as well as regular expressions to search through file contents of head as well as earlier revisions of files stored on the server. While not every single option of a standard grep is supported, the most important options are available. Here is the syntax of the command according to ‘p4 help grep’:

p4 grep [ -a -i -n -v -A after -B before -C context -l -L -t -s -F -G ] -e pattern file[revRange]...

Most users of the standard grep will be used to the syntax ‘grep pattern files…’, but the format ‘grep -e pattern files…’ is a valid alternative syntax useful if the pattern contains a ‘-’ character. For ‘p4 grep’ the ‘-e pattern’ syntax was chosen to easily distinguish the pattern from the file arguments.

Usage examples

Instead of explaining every option, which you might know from the standard grep already (or can look up in the documentation), I would like to demonstrate some use cases to show how ‘p4 grep’ could be used.

Fixed strings

I am looking for the string ‘compile_settings’ in Jam’s source code:

p4 grep -e compile_settings Jam/MAIN/src/...

This will return all lines of all files of the head revision that contain this phrase, including the revision number of each file:

//depot/Jam/MAIN/src/compile.c#35: *    compile_settings() - compile the "on =" (set variable on exec) statement
//depot/Jam/MAIN/src/compile.c#35: * compile_settings() - compile the "on =" (set variable on exec) statement
//depot/Jam/MAIN/src/compile.c#35:compile_settings( parse, args )
//depot/Jam/MAIN/src/compile.h#10:void compile_settings();
//depot/Jam/MAIN/src/jamgram.c#21:# define pset1( l,p,a )      parse_make( compile_settings,p,P0,S0,S0,l,L0,a )
//depot/Jam/MAIN/src/jamgram.y#14:# define pset1( l,p,a )      parse_make( compile_settings,p,P0,S0,S0,l,L0,a )
//depot/Jam/MAIN/src/jamgram.yy#16:# define pset1( l,p,a )      parse_make( compile_settings,p,P0,S0,S0,l,L0,a )

‘p4 grep’ uses the same logic as ‘p4 files’ or ‘p4 print’: the file arguments can be adorned with a revision range. ‘p4 grep’ will then search the file with the highest available revision in that range, unless the option ‘-a’ is used, in which case every revision in that range is considered.

Read More

One of the duties of a branch curator (usually a senior developer or project manager) is periodically refreshing her branch with changes from an upstream branch. For example, if she is curating a development branch, she will need to periodically pick up changes from the MAIN or INTEGRATION branch.

When I’ve had this responsibility, I constantly looked for ways to remind myself to run the integrations frequently. I really wanted to see a list of pending integrations in a tool that I used every day. I’ve tried using widgets in wikis and intranets, scripts that sent pending integrations weekly via email, and a few other methods.

Now, with the 2010.1 beta release, there’s a better way. Using the Perforce Javascript API (P4JsApi) (PDF), we can put our own tabs (applets) in P4V, which is the tool I always have open when I work in Perforce.

In order to create a Branch Curator’s Dashboard, let’s start by making a Pending Integrations tab (applet). This tab will show me, for each branch spec that I own, all pending integrations, using the forward and reverse view mapping. (I always use branch specs to manage integration activities.)

The source code and a README file with more complete installation instructions are available in the Public Depot. Here’s a brief description of how the tab works.

function settings(key) {
    if (key == "p4v_mainTabs")  {
        return ["p4:///files/depot/jsapi/interchanges.html"];
    }

}

list group BranchCurators.G centralsettings
    //depot/jsapi/centralsettings.js

// get all branch specs owned by current user
var branches = P4JsApi.p4("branches -u " + P4JsApi.getUser());
for (var i = 0; i < branches.size; i++) {
    var branch = branches.data[i];

    // get view
    var branchspec = P4JsApi.p4("branch -o " + branch.Branch);
    var origview = branchspec.data[0].View0;
    var indexofdiv = origview.lastIndexOf("//");
    var view = P4JsApi.encodeForHTML(origview.substring(0,indexofdiv))
        + "<br>   "
        + P4JsApi.encodeForHTML(origview.substring(indexofdiv));

    // get pending forward integs
    var fpending = P4JsApi.p4("integrate -n -b " + branch.Branch);
    var fpendinglist = '';
    for (var j = 0; j < fpending.size; j++) {
        fpendinglist = fpendinglist
            + P4JsApi.encodeForHTML(fpending.data[j].depotFile)
            + "<br>";
    } // all forward pending integs

    // get pending reverse integs
    var rpending = P4JsApi.p4("integrate -n -r -b " + branch.Branch);
    var rpendinglist = '';
    for (var j = 0; j < rpending.size; j++) {
        rpendinglist = rpendinglist
            + P4JsApi.encodeForHTML(rpending.data[j].depotFile)
            + "<br>";
    } // all reverse pending integs

    // add branch to table
    // ... use HTML DOM to insert rows ...

Save a tree! Don't actually print it!

When getting started with Perforce or expanding its scope within an enterprise, it’s a good time to establish a formal, documented Perforce Directory Standard (PDS).  The directory structure and branching strategy are intimately related with Perforce. Branching strategy and product life cycle are inherently related.  A well-designed directory structure intuitively conveys information about product life cycle, the release process, and branching strategy.  Further, simplifies management of the flow of change.  There’s a lot going on there!

There is a fairly canned process I follow to help a customer define and capture their branching strategy.  First, I try to get everyone speaking the same language, defining terms such as “branching patterns,” “branch,” “merge down,” “copy up,” etc.  Next, I provide a brief overview of common branching patterns, to give an idea of the molds available.

Then we discuss either their current processes or the process they’d like to evolve to.  By the end of that discussion, we have an initial idea of which branching patterns apply to their situation.  Most often I see new samples of familiar patterns.  In rare cases, I come across a customer with an environment that needs a new mold.  But I’ve been at this game a while, and those are few and far between these days.  If you think you have one of those and would like to share, by all means, please do!

Yikes! Not another CM Audit!

In addition to their intrinsic value, a PDS is handy to have in the event of a Configuration Management audit.  Trying to climb the SEI CMMI scale? Being audited during contractual due diligence? This helps!

The PDS stands on its own, and can be incorporated into a large document hierarchy.  It’s best deployed as a living document that evolves as your organization’s needs change.

If you’re interested in the template, let me know.  Disclaimer:  It’s forever a work in progress, and your mileage may vary as to its applicability to your environment.

Home

• About

  • p4 blog
  • Disclaimer

• Categories

  • Administration
    • Disaster Recovery
    • High Availability
  • Agile
  • ALM
  • Architecture
  • Best practices
    • Security
  • Branching
  • Branching Strategy
  • Certification
  • Cloud Computing
  • Collaboration
  • Consulting
    • Automated Lifecycle Management
    • Consultant's Toolkit
  • Documentation
  • Eclipse
  • Events
    • User Conference 2009
    • User Conference 2010
  • Java
  • JavaScript API
  • Mashups
  • Migrations
  • Multi-platform
  • New features
  • P4V
  • Performance
  • Philosophy
  • Programming
    • C++ API
  • Releases
  • Reporting
  • Scripting
  • Server
  • Soapbox
  • Social media
  • Tips and Tricks
  • Training
  • UI Design
  • Uncategorized
  • visualization
  • Web Services

• Archives

  • September 2010
  • August 2010
  • July 2010
  • June 2010
  • May 2010
  • April 2010
  • March 2010
  • February 2010
  • January 2010
  • December 2009
  • November 2009
  • October 2009

• Authors

  • Alan Teague (1)
  • Bill Baffy (1)
  • Dave Robertson (4)
  • Dan Steele (1)
  • James Creasy (3)
  • Jason Novecosky (1)
  • Jason Gibson (16)
  • Julia Angell (2)
  • Kevin Sawicki (5)
  • Laura Wingerd (2)
  • Marc Tooley (1)
  • Marc Wensauer (1)
  • Matt Attaway (1)
  • Michael Bishop (1)
  • Nellie Lemonier (1)
  • Randall DeFauw (6)
  • Rachael Fish (3)
  • Rusty Jackson (7)
  • Sheri Aguirre (3)
  • Sam Stafford (1)
  • Stewart Lord (3)
  • Sven Erik Knop (13)
  • Tony Corman (2)
  • Tony Smith (3)
  • Tony Vinayak (5)
  • Tom Tyler (18)
  • Bill White (1)