Showing posts with label Data Migration. Show all posts
Showing posts with label Data Migration. Show all posts

Wednesday, August 29, 2012

Data Migration - "Created By" Field Nightmare

As Sharepoint consultants we have often been faced with this opportunity to port the data across environments. I came across such a scenario in one of my recent engagements where I had to push all the data from Excel to Sharepoint Lists.

Below I used data migration from excel sheet, but if any migration tool that you have used does not support for the migration of created by and created date fields - you can fine tune the below code to your benefit.
 

Couple of issues I would like to address which I faced and the solutions to those to help my SharePoint community members.

1. Import n excel sheet to create a List via the Sharepoint User Interface - This has worked fine except that the look ups were not carried through in the List. I had to update the column types after the create.


 
 
 
 

 
 
 
 



 
The look up column for Country needed to be changed
 
 
 
Note : If the Excel column for Country did not have values from the data validation list, then you will need to add those values in the choice column above. By default it takes all the values form the Sharepoint column and converts it to a choice. 
 
 
2.  "Created by" fields were mirrored with admin account. I needed to now convert these to the users who had created them since there was functionality to display only your own items to contributors.
 




Here is a Visual web part I came up with to update the "Created by" and "Modified by" field to any other user based on an additional column value - "CopyCreatedBy" to update the user name. 
 
 
 
For entering data - I could not view the List in data sheet view, Some research on the net showed I needed to install this below.
If you do not see the data sheet view - Install the update for X64, to access Data sheet view. 2007 Office System Driver: Data Connectivity Components
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=23734
OK, That sorted that issue out. Now I entered some data for the user names.
 
 
 
 
UX:
 
 
 
 
 The below code is not permormance savvy since it uses the crude way to access the list & data.
Although the code works fine and performs great in a local environment, issue is with the way the Items property is accessed. The Items property queries ALL items from the Content Database for the current SPList and “unfortunately” does that every time we access the Items property. The retrieved items ARE NEVER CACHED. In the loop the Items property is accessed twice for every loop iteration – once to retrieve the Count, and once to access the actual Item identified by its index.
 
Code:
 using System;  
 using System.Web.UI;  
 using System.Web.UI.WebControls;  
 using System.Web.UI.WebControls.WebParts;  
 using Microsoft.SharePoint;  
 namespace ChangeUserName.ChangeCreatedBy  
 {  
   public partial class ChangeCreatedByUserControl : UserControl  
   {  
     protected void Page_Load(object sender, EventArgs e)  
     {  
       if (!IsPostBack)  
       {  
         lblerror.Visible = false;  
         txtListName.Text = "";  
         txtwebSite.Text = "";  
         txtCopyCreateByCol.Text = "";  
       }  
     }  
     protected bool validate()  
     {  
       if (txtListName.Text =="" ||txtwebSite.Text == "" || txtCopyCreateByCol.Text == "")  
       {  
         lblerror.Visible = true;  
         lblerror.Text += "Please enter the Required Fields with *" ;  
         return false;  
       }  
       return true ;  
     }  
     protected void Update_Click(object sender, EventArgs e)  
     {  
       if (validate())  
       {  
         lblerror.Text = "";  
         try  
         {  
           using (SPSite site = new SPSite(txtwebSite.Text))  
           {  
             using (SPWeb web = site.OpenWeb())  
             {  
               SPList list = web.Lists[txtListName.Text];  
               foreach (SPListItem item in list.Items)  
               {  
                 web.AllowUnsafeUpdates = true;  
                 SPUser user = web.EnsureUser(item[txtCopyCreateByCol.Text].ToString());  
                 string value1 = user.ID + ";#" + user.Name;  
                 item["Author"] = value1;  
                 item["Editor"] = value1;  
                 item.Update();  
               }  
               list.Update();  
               web.Update();  
               lblerror.Visible = true;  
               lblerror.Text = "Successfully updated the created by field for all records of the list : " + txtListName.Text;  
             }  
           }  
         }  
         catch (Exception ex)  
         {  
           lblerror.Visible = true;  
           lblerror.Text += "All Records could not be updated. " + ex.Message.ToString();  
         }  
       }  
     }  
   }  
 }  

Monday, July 30, 2012

Migrating File store to SP2010 ECM considerations

Migration Considerations


During the planning process of a migration, decisions should be made early as to whether content is migrated directly into SharePoint 2010 "as - is" or whether this opportunity should be explored to make any changes to the information architecture, archiving, or look and feel and address the pain points. Since, making changes to any of these areas can complicate the migration process, some organizations choose to focus simply on migrating the content to SharePoint 2010, and then enhancing the other aspects of content are pushed to the later phases of the project. Simplifying the migration can help to reduce the coefficient of risk by substantially reducing the number of variables & at the same time reorganizing the content and make minor changes to the look and feel of may go a long way in driving adoption.
Other organizations choose to take the opportunity of the migration to organize their content in a better way at the same time as it is moved into SharePoint 2010. This may involve updating the information architecture, revamping security, implementing archiving, storage optimization and retention policies, or consolidating and reorganizing the content.

 
If the migration is a Sharepoint to sharepoint migration, adoption is less of an issue later, since the users are already used to a certain way of accessing files which is not drastically changeing after the upgrade process.

However, if this is a File server to SharePoint migration we need to be careful since the access and process of using the file has changed somewhat and there is a learning curve required by the end user to successfully use the document management features to be most productive.

From my experience, here's a list of supporting points to use  the document management :
  • Document Library :
    Document Library is a List template used to create a repository of documents to be accessed by the users. This is one of the parts of a site which has other tools available for collaboration. A document library  is a specialized list with functionality tailored to managing
    documents.
  • Document Workspace:
    This should be used when there is a team that wants to collaborate on a few documents  say < 5,000 or so. This serves the purpose without the overhead of maintenance and storage. A workspace is a specialized type of site with functionality tailored around either a single document (a document workspace) or a scheduled meeting (a meeting workspace).
  • Document Center:
    This is a site template dedicated to manage a large volume of active & changing documents. Any time a document can be declared as a record by either moving it to a Record Center or declaring the document as a record in same document library using the in-place feature.The Document Center site template supports creating knowledge base archives. Typically, knowledge bases contain single versions of documents, and a site can scale to 10 million files. In a typical scenario, such as a technical support center for a large organization, 10,000 users might access the content, primarily to read it. A subset of 3,000 to 4,000 users might upload new content to the site
  • Record Center :
    This is a site template specifically dedicated for managing large volume of records (documents that are not modified),Tens of millions of records in a single Records Center, Hundreds of millions of records in a distributed archiveThis is an effective place for Hierarchy, Driven By Metadata 
     Here are some of the major features:
    • Document ID: Every document can be assigned a unique identifier, which stays with the document even when it's archived. This allows records to be easily referenced by an ID no matter where the document moves.
    • Multi-Stage Retention: Retention policies can have multiple stages, allowing you to specify the entire document life cycle as one policy (e.g. review Contracts every year, and delete after 7 years)
    • Per-Item Audit Reports: You can generate a customized audit report about an individual record.
    • Hierarchical File Plans: You can create deep, hierarchical folder structures and manage retention at each folder in the hierarchy (or inherit from parent folders).
    • File Plan Report: You can generate status reports showing the number of items in each stage of the file plan, along with a roll up of the retention policies on each node in the plan.
    • Taxonomy and Centralized Content Types: The archive will be a consumer of enterprise-wide taxonomies and content types, ensuring consistency and context transfer between the collaborative spaces and the archive. We'll be talking a lot more about our 2010 taxonomy investments in future posts.
    • Content Organizer: The records router can use meta data to route incoming documents to the right place in the hierarchical file plan. For instance, it enables you to automatically enforce rules on content that is submitted, like "If a Purchase Agreement is tagged with Project Alpha, send to the Alpha Contracts sub folder and apply that's folder retention policy to the item."
    • Virtual Folders: The file plan is a great way to manage a repository but often time isn't what you want to use to navigate and find the content you are looking for. The SharePoint 2010 Records Center makes use of a new feature called meta data based navigation, which allows you to expose key meta data as virtual folders:

Friday, June 8, 2012

Data Migration to the cloud!

Data Migration from a third party application to Sharepoint is often a requirement and I had this challenge on the Sharepoint online environment, where we do not have access to the Central Admin or the server itself.

Where there is a will there is a way! We had a challenging situation and we figured out a workaround to do the needful...

Since it was Lotus application data- we needed a means to get the data in a Microsoft format so that we could read it ( any third party connections are not supported in the cloud yet), Lotus connectors are not supported.

Solution : get all the data ported to an excel format which can be read from Sharepoint environment!
The data includes attachments as well - so design your excel in such a manner that the last 2 columns have the below information:


  1. Number of attachments
  2. Path and Name of attachments in the format ( c://attachments/recordId/test1.docx)

This uses the sharepoint client object model to upload the data and since the attachments behaves a little sneaky I used the web services to upload the attachments once the record was created!
There are enough comments in the code to explain the logic but in case you have any questions please let me know!

This console application needs to run in the same domain for picking up Admin credentials of the logged ion user for the upload process and takes a config file which details the parameters like, site, list name etc. Add a web Reference to the
http://localhost/_vti_bin/Lists.asmx and call it "SharepointListWS"

Here is the sample excel format : The second hidden row of the excel is the sharepoint column names from the List where the data is uploaded to.

Here is the code snippet that I created to import data from Lotus apps ( in a Excel format) to SPO world.

   
  
  
  private static void importFromExcel(StreamWriter streamwriterW)
  {
  
  try
  
 {
  
 Microsoft.Office.Interop.Excel.Application excelApp = new Microsoft.Office.Interop.Excel.ApplicationClass();
  
 string workbookPath = _ExcelPath;
  
 Microsoft.Office.Interop.Excel.Workbook excelWorkbook = excelApp.Workbooks.Open(workbookPath, 0, false, 5,
  
 "", "", false, Microsoft.Office.Interop.Excel.XlPlatform.xlWindows, "", true, false, 0, true);
  
 Microsoft.Office.Interop.Excel.Sheets excelSheets = excelWorkbook.Worksheets;
  
 Microsoft.Office.Interop.Excel.Worksheet excelSheet = (Microsoft.Office.Interop.Excel.Worksheet)excelSheets.get_Item("Sheet1");
  
 object[,] values = excelSheet.get_Range(_ExcelRange).Value as object[,];
  
 ClientContext context = new ClientContext(_SharePointSiteUrl);
  
 context.AuthenticationMode = ClientAuthenticationMode.Default;
  
 Web web = context.Web;
  
 List list = web.Lists.GetByTitle(_ListName);
  
 for (int row = 3; row <= Convert.ToInt32(_ExcelRowNo); row++) // no. of rows of data in excel
  
 {
  
  ListItemCreationInformation listItemCreation = new ListItemCreationInformation();
  
 ListItem listItem = list.AddItem(listItemCreation);
  
 iRecord++;
  
 for (int column = 1; column <= Convert.ToInt32(_iColumnsToImport) + 1; column++) // no of fields/columns in excel to be imported
  
 {
  
 string columnName = values[2, column].ToString();
  
 strcolumnName = columnName;
  
 if (values[row, column] != null)
  
 {
  
  if (columnName == "AttachmentNo")
  
 {
  
 _strNoOfAttachments = values[row, column].ToString();
  
 strcolumnValue = _strNoOfAttachments;
  
 }
  
  else
  
 {
  
 listItem[columnName] = values[row, column];
  
 strcolumnValue = listItem[columnName].ToString(); 
  
 }
  
 }
  
 }
  
 listItem.Update();
  
 context.ExecuteQuery();
  
 string strListItemID = listItem.Id.ToString();
  
 _NoOfRowsImported++;
  
 
 logToFile("Record imported from Excel Row No :" + (iRecord + 2) + " to SP Record Id : " + strListItemID, streamwriterW);
  
 logToFile("No. of Attachments : " + _strNoOfAttachments, streamwriterW);
  
   if (_strNoOfAttachments != "0" && _strNoOfAttachments != null)
  
 {
  
 addAttachment(strListItemID, values[row, Convert.ToInt32(_iColumnsToImport) + 2].ToString(), streamwriterW);// if no. of attachment >0 
  
 }
  
 }
  
 excelWorkbook.Close();
  
 }
  
  catch (Exception e)
  
 {
  
 // log error here 
  
 }
  
 }
  
  private static void addAttachment(string strNewListItemID, string strAttachmentPathUnparsed, StreamWriter streamwriterW)
  
 {
  
  string attachmentUrl = string.Empty;
  
 try
  
 {
  
 //Parse path if delimiter exists 
  
 string[] splitAttachmentPath = strAttachmentPathUnparsed.Split(new Char[] { '|' });
  
 foreach (string strAttachmentPathParsed in splitAttachmentPath)
  
 {
  
  if (strAttachmentPathParsed.Trim() != "")
  
 {
  
 strAttachmentPath = strAttachmentPathParsed;
  
  if (!System.IO.File.Exists(strAttachmentPathParsed))
  
 {
  
 // log error

   return;
  
 }
  
  string fileName = Path.GetFileName(strAttachmentPathParsed);
  
 // Get file contents
  
 Byte[] contents = getFileStream(strAttachmentPathParsed);
  
 // Add the attachment to the new item in the SharePoint list & Instantiate a SharePoint
  
 // List Web Service object
  
 listWs = new SharePointListWS.Lists();
  
 // set the Web Service Credentials
  
 listWs.Credentials = System.Net.CredentialCache.DefaultCredentials;
  
 listWs.Url = _SharePointSiteUrl + "/_vti_bin/lists.asmx";
  
 attachmentUrl = listWs.AddAttachment(_ListName, strNewListItemID, fileName, contents);
  
 }
  
 }
  
    
 }
  
  catch (Exception ex3)
  
 {
  
 //log error  
 }
  
  if (!String.IsNullOrEmpty(attachmentUrl))
  
 {
  
     // log error

 }
  
 }
  
 }
  
 }