Vendor access to Database - Not a Good Idea

Recently, I was queried on what would be the best way to retrieve data from a database within our enterprise, which is required by a vendor to consume. Basically, an external application looking to consume our data.
    As obviously, suggested to expose as API (REST/SOAP) or spew out a CSV/TXT/sth. file which will be consumed by the external vendor. It seems that none of the above could work for the vendor, without writing some customized code. With this stage set, I thought of enumerating why one shouldn't give access to their database to someone whom we aren't in control. Let's begin !

(1) Accessing data through a client application/DB Connectivity API (Eg. JDBC)
NO-NO:   Database patches/upgrades are normal and if we have to do any of those, we need to remind ourselves to check with the external vendor, whether it is all good, from their DB access API/client. This could have a roll-on effect on other databases installed in the same host. A very tight dependency, which DBAs will hate.
      Additionally, there is no assurance of how good the database connectivity life-cycle is. Personally seen external applications screwing up enterprise database, by not closing DB connections or utilizing a high DB processes, which in turn affect other applications performance. The very hard part is debugging the issue, when all that one knows an external application "ABC" is accessing our database.

(2) Giving access as a database view, to prevent any misadventures from external vendor.
NO-NO:  Consider a situation, where we would like to use the same DB view with some minor datatype or new field changes, we need to go to the vendor to check whether this is fine. If not, we will create another database view, very similar to the one used by the external vendor and the schema mess-up begins here.


(3) Vendor says, direct DB access is the only way
NO-NO:  This is where the principles of integration within organization kick-in. Deviating from it, means we are getting into the "spaghetti" world of integration, which may not be a nightmare tonight, but will happen for sure in the near future.

Can microservices simplify SQL queries?

I have seen very complex SQL queries embedded within code, esp. when they are across business applications boundaries. Eg. Finance, Billing, GIS SQL queries, working together to retrieve data of interest. Disparate systems making it to work as "one",
   Can microservices bring simplicity to that, as it is the very nature of it to implement them in a bounded specific domain-level context ?  It sounds logical to me that the answer is "YES"!
Again, someone for the love/hate of it can make a complex SQL or by the very nature of the complexity of the underlying data relationship. 

NoESB - A Marketing Gimmick ?!

Some API vendors are doing a peaceful web-oriented fighting with some ESB vendors. The core content is, ESB is not required or better don't use it if you want to improve your time to market and making the organizational software foundation agile.
   Well, well, all those arguments I see are related to cloud and new products/development. The interesting part is putting forward that API Gateway is a replacement for the ESB, where the API Gateway does the similar functionality to ESB. Now this argument itself debunks the theory of "NoESB", when you have subset of similar functionality in 2 different approaches.
   Did some more Googling and landed up at WSO2 site - a popular API vendor, not that I am fan of it. In SlideShare API-Centric Architecture  they have a clear point on that their API gateway is actually a full-blown ESB with some constraint and one has the choice to install the missing features to make it a ESB !
   ESB has their role as much as API, like for integrating in complex scenarios, file/ftp based and legacy application integrations. At the same time, I am a strong supporter on "smart ends and dumb pipes", which I have many times seen violated in BPEL processes. Business logic should be encapsulated in its domain and not be floated in a middleware. This is where the µservices principle stands attractive.
   It is a pity to see vendors(again!) playing their marketing games and continuing with the confusion era as much as it started with SOA. Beware is the only word, I say to myself !

  

Journey of Integration

The usual Initial Ramblings
            In my new job I am supposedly THE person in getting an integrated integration solution(along with external consultants) in place and thereby lend for future business process automation. Prior to this one, I was working in Oracle SOA in BPEL-JDeveloper combo. and all was honky dory - basically all where happy and ended/continued in peace and love !  This is not the case now - good to keep me ticking in my integration knowledge pursuit and bad when you see so many difference techniques, approaches, thoughts and more importantly our wonderful vendors pursuit to keep it confusing - may or may not be deliberate but obviously for monetary purposes.

Let's Take Off !

Been reading lots of articles, some books and watching my mind in putting together all that was read. Simply put, a lot of stuff and very little conclusion with more towards confusion. Let us go through the list of magic words - SOA, BPEL, ESB, EDA, Messaging, Integration frameworks, Middleware, Process Orchestration, Process Choreography, Process Workflow, BPM, BAM, Queues, Microservices, REST, API and the endless list continues.

      Statements like this "Design and publish APIs for end customers, B2B, or mobile - APIs can be in SOA or REST" make things more a hell. This is supposedly from integration leaders' product documentation. APIs can be in SOA ?!  SOA is an architectural principle and SOA can be pursued through SOAP over HTTP (that is what the doc. is probably trying to say), REST style services, Microservices and anything more some integration gurus can add to.
    Since we touched upon SOA, let us probably look at the different approaches of SOA. The technical gurus prefer the word "Service Oriented Approach" instead of "Service Oriented Architecture", which seems from history (when I was physically a kid !), a failure, across the geography.

From my perspective, SOA could be implemented as "BPEL" or "ESB".  The choice of which one depends on the needs, which makes sense isn't it ?! Hope you are part off the minority!

•  BPEL is fine for long-term processes and involves human interaction. Due to the language nature, it is heavy and can involve complex logic, including business ones. This is where it may not suit the new fad of get SOA the right way - Mrs. Microservices, where you want to keep the smarts at the end points and not in the middleware.
• ESB is a mechanism to move messages supposedly in a light weight manner with minimal required logic/actions to the message within the bus/pipe. This may fit the Mr. Microservices design style. The other denouncing term is "Egregious Spaghetti Box". So watch out what you are trying to achieve. Keep you principles clear and focused. 

Hope I haven't added another mess to the existing mess in the integration world.

Great links for Maps

Coloring
Going through some links but can't stop appreciating this nice online app. for choosing color combination while authoring maps - Color Brewer. If someone knows of anything more please do drop in a comment.

Symbols

• Check-out MapBox MAKI to download set of commonly used symbols while creating point objects.
• Penn State Uni  Symbol Store. It includes a nice way to prepare the symbol "AFTER" adding to the cart.

ERP and Maps

Long-time since I blogged something....lots of changes from then to now!
Different job(can't say new!), different positions and ofcourse new kids as a bonus :-)
  I have been doubling up as a Spatial Systems Co-ordinator along with my SOA integration tasks, the former being in simple terms taking care off map-based applications and any data related to it. Very interesting and a useful one for the business and end-users. In that way proud of my efforts. Off recently we are implementing a ERP solution and me being part off the SOA integration team has brought in a new perspective in terms of mapping and integration solutions.
   

jqGrid with ASP .NET Web Forms

A quick write-up on my findings to use the amazing jqGrid v3.4.3 with ASP .NET 2.0 Web Forms. Surprisingly, there is no clear demo./website which enumerates on this usage. After some effort, got it finally working. Please note this is w.r.to ASP .NET 2.0 and I am just a day/two old with jqGrid!

(1) Suggested to not use PageMethods - can look at the "different" pains by different people at this link - http://www.west-wind.com/weblog/posts/152493.aspx.
If needed I can add mine into the list!

(2) The relevant partial code in the *.aspx file(without sending any parameters) is:

.......
.......
jQuery(document).ready(function() {
$('#list').jqGrid({
datatype: function() {
$.ajax({
url: "jqGrid.asmx/jQGridDataASMX",
data: "{}",
dataType: "json",
type: "POST",
contentType: "application/json; charset=utf-8",
complete: function(jsondata, stat) {
window.alert("Status received is " + stat);
window.alert("Response text is: " + jsondata.responseText);
if (stat == "success") {
var thegrid = jQuery("#list")[0];
thegrid.addJSONData(eval("(" + jsondata.responseText + ")"));
}
}
});
},
colNames: ['Col1', 'Col2', 'Col3'],
colModel: [
{ name: 'Col1', index: 'Col1', width: 40, align: 'left' },
{ name: 'Col2', index: 'Col2', width: 40, align: 'left' },
{ name: 'Col3', index: 'col3', width: 200, align: 'left'}],
pager: jQuery('#pager'),
rowNum: 10,
rowList: [10, 20, 30],
viewrecords: true,
imgpath: 'jqGrid-3.4.3/themes/basic/images',
caption: 'jqGrid NewComer - First Grid'
});
});


........
........

When sending parameters to the web service method use the JSON object which is defined in "json2.js"(You can download from - http://www.JSON.org/json2.js).
Eg. var jsonStr = JSON.stringify(data);
and then place the jsonStr variable within the "data" parameter of the $.ajax request like:
url: "jqGrid.asmx/jQGridDataASMX",
data: jsonStr,
dataType: "json",


(3) The complete jqGrid.asmx file is:

using System;
using System.Collections.Generic;
using System.Web;
using System.Web.Services;
using System.Web.Script.Services;
using System.Web.Script.Serialization;

public class RowElement
{
public string id;
public string[] cell;
}

public class MainGrid
{
public string total;
public string page;
public string records;
public RowElement[] rows;
}


///

/// Summary description for jqGrid
///

[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[ScriptService]
public class jqGrid : System.Web.Services.WebService {

[WebMethod]
public MainGrid jQGridDataASMX()
{
/* {
total: 'xxx',
page: 'yyy',
records: 'zzz',
rows : [
{id:'1', cell:['cell11', 'cell12', 'cell13']},
{id:'2', cell:['cell21', 'cell22', 'cell23']}
]
} */

MainGrid mg = new MainGrid();
mg.total = "1";
mg.page = "1";
mg.records = "1";

mg.rows = new RowElement[1];
mg.rows[0] = new RowElement();
mg.rows[0].id = "1";
mg.rows[0].cell = new string[3];
mg.rows[0].cell[0] = "cell1";
mg.rows[0].cell[1] = "cell2";
mg.rows[0].cell[2] = "cell3";

return mg;
}

}


(5) The resultant output is:

Hope this helps for someone to get a heads-up with jqGrid and ASP .NET 2.0!


Also check-out a very good site on the jQGrid with ASP .NET using HTTP handlers - http://geeks.netindonesia.net/blogs/cipto/archive/2009/04/03/jqgrid.aspx